# Loadig the LLAMA 30B Model. Memory Issue

**URL:** <https://discuss.huggingface.co/t/loadig-the-llama-30b-model-memory-issue/38357>\
**Category:** Models\
**Created:** [May 2, 2023, 1:09am UTC](https://discuss.huggingface.co/t/loadig-the-llama-30b-model-memory-issue/38357 "2023-05-02T01:09:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pial08](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@pial08](https://discuss.huggingface.co/u/pial08)\
**Post date:** [May 2, 2023, 1:09am UTC](https://discuss.huggingface.co/t/loadig-the-llama-30b-model-memory-issue/38357/1 "2023-05-02T01:09:10Z")

</div>

Hi,  
I am trying to load the LLAMA 30B model for my research. I already downloaded it from meta, converted it to HF weights using code from HF. However, I tried to load the model, using the following code:

```auto
model = transformers.AutoModelForCausalLM.from_pretrained(

        model_args.model_name_or_path,

        cache_dir=training_args.cache_dir,

    )

```

I get the following error:

```auto
Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36704 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36706 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36707 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36708 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36710 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36712 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 36715 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 1 (pid: 36705) of binary: /usr/bin/python
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.1.0a0+fe05266', 'console_scripts', 'torchrun')())
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/ __init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

```

During model loading, it uses almost 1005GB of memory.

 ![MicrosoftTeams-image (3)](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9401da360aa5217880c0520f23d075b56ad64108.jpeg)

I loaded the 7B model earlier, which worked fine.  
Is the issue that, since it’s a extremely large model, my DGX cannot handle with its 1005 GB of memory or I am missing something here.

I would really appreciate any suggestions.

---

<div class="post-metadata">

**Author:** ![anxiangyinyun](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@anxiangyinyun](https://discuss.huggingface.co/u/anxiangyinyun)\
**Post date:** [July 26, 2023, 8:53am UTC](https://discuss.huggingface.co/t/loadig-the-llama-30b-model-memory-issue/38357/2 "2023-07-26T08:53:36Z")

</div>

Hi, I had the same OOM trouble when loading Llama2-70b,  
Have you solved this problem?  
I would appreciate your suggestions.

---

<div class="post-metadata">

**Author:** ![pial08](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@pial08](https://discuss.huggingface.co/u/pial08)\
**Post date:** [July 27, 2023, 8:14am UTC](https://discuss.huggingface.co/t/loadig-the-llama-30b-model-memory-issue/38357/3 "2023-07-27T08:14:24Z")

</div>

I was able to solve the problem.

I was using torchrun to load the model and the value of nproc\_per\_node was 8.  
So the system was loading the model, it was creating 8 instance of the same model and essentially crashing the program because of low memory.

If this is the case for you, you can try setting the value to 4 or 2 and see what happens.

Let me know if this solves the issue.