# Starcoder: CUDA out of memory

**URL:** <https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765>\
**Category:** Beginners\
**Created:** [November 27, 2023, 10:24pm UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765 "2023-11-27T22:24:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shrey-shah](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@shrey-shah](https://discuss.huggingface.co/u/shrey-shah)\
**Post date:** [November 27, 2023, 10:24pm UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765/1 "2023-11-27T22:24:46Z")

</div>

I am trying to run

> bigcode/starcoder

model on Amazon EC2. I have 8 tesla T4 GPUs with 16GB RAM, but somehow I still encounter the _“Cuda out of memory”_ error.

What possible solutions can this issue have? Thank you.

Code Snapshot:

> from transformers import AutoModelForCausalLM, AutoTokenizer  
> from accelerate import Accelerator  
> accelerator = Accelerator()  
> checkpoint = “bigcode/starcoder”  
> device = accelerator.device # for GPU usage or “cpu” for CPU usage  
> print(‘Reached device Selection.’)  
> tokenizer = AutoTokenizer.from\_pretrained(checkpoint)  
> print(‘Tokens generated.’)  
> model = AutoModelForCausalLM.from\_pretrained(checkpoint).to(device)  
> accelerator.prepare(model)  
> inputs = tokenizer.encode(“def print\_hello\_world():”, return\_tensors=“pt”).to(device)  
> outputs = model.generate(inputs)  
> print(tokenizer.decode(outputs[0]))

Error:

 ![Screenshot 2023-11-27 154125](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/7/47f1d8c09cad21a3afea2415472a0b11acf4e82a.png)

---

<div class="post-metadata">

**Author:** ![lkthomas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lkthomas/32/23687_2.png) [@lkthomas](https://discuss.huggingface.co/u/lkthomas)\
**Post date:** [February 9, 2024, 6:02am UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765/2 "2024-02-09T06:02:09Z")

</div>

same issue here

---

<div class="post-metadata">

**Author:** ![CKeibel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckeibel/32/55416_2.png) [@CKeibel](https://discuss.huggingface.co/u/CKeibel)\
**Post date:** [February 9, 2024, 7:41am UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765/3 "2024-02-09T07:41:53Z")

</div>

Hi!  
I searched the internet and it seems that `accelerator.device` always uses `cuda:0`, so the model is not distributed to the GPUs.

Maybe [this](https://discuss.huggingface.co/t/how-to-use-specific-gpu-in-accelerate/62810) helps you.

---

<div class="post-metadata">

**Author:** ![lkthomas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lkthomas/32/23687_2.png) [@lkthomas](https://discuss.huggingface.co/u/lkthomas)\
**Post date:** [February 22, 2024, 9:28am UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765/4 "2024-02-22T09:28:51Z")

</div>

is it a limit for most of the LLM model which can’t scale into multiple GPUs?

---

<div class="post-metadata">

**Author:** ![Zahcoder34](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zahcoder34/32/24165_2.png) [@Zahcoder34](https://discuss.huggingface.co/u/Zahcoder34)\
**Post date:** [February 23, 2024, 4:44pm UTC](https://discuss.huggingface.co/t/starcoder-cuda-out-of-memory/63765/5 "2024-02-23T16:44:01Z")

</div>

> [@shrey-shah](#):
>
> model = AutoModelForCausalLM.from\_pretrained(checkpoint).to(device)

AutoModelForCausalLM.from\_pretrained has device\_map option, set it to `device_map="auto"` it will split model layers into different available devices.

Correct:  
`model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto").to(device)`
