# ValueError: weight is on the meta device when using Auto Model For Sequence Classification

**URL:** <https://discuss.huggingface.co/t/valueerror-weight-is-on-the-meta-device-when-using-auto-model-for-sequence-classification/58653>\
**Category:** 🤗Accelerate\
**Created:** [October 15, 2023, 6:47pm UTC](https://discuss.huggingface.co/t/valueerror-weight-is-on-the-meta-device-when-using-auto-model-for-sequence-classification/58653 "2023-10-15T18:47:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Glick](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@Glick](https://discuss.huggingface.co/u/Glick)\
**Post date:** [October 15, 2023, 6:47pm UTC](https://discuss.huggingface.co/t/valueerror-weight-is-on-the-meta-device-when-using-auto-model-for-sequence-classification/58653/1 "2023-10-15T18:47:37Z")

</div>

Trying to load Llama for classification, I get a ‘weight is on the meta device’ error.  
Code:

```auto
model = AutoModelForSequenceClassification.from_pretrained('decapoda-research/llama-7b-hf', torch_dtype=torch.float16, device_map="auto", num_labels=3)

```

I get the error:

```auto
model = AutoModelForSequenceClassification.from_pretrained( File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 471,  
 in from_pretrained return model_class.from_pretrained( File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2846,   
in from_pretrained dispatch_model(model, device_map=device_map, offload_dir=offload_folder, offload_index=offload_index)   
File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/big_modeling.py", line 396,   
in dispatch_model attach_align_device_hook_on_blocks(   
File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/hooks.py", line 537,  
 in attach_align_device_hook_on_blocks attach_align_device_hook_on_blocks( File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/hooks.py", line 507,  
 in attach_align_device_hook_on_blocks add_hook_to_module(module, hook) File  
 "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/hooks.py", line 155,  
 in add_hook_to_module module = hook.init_hook(module) File  
 "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/hooks.py", line 253,   
in init_hook set_module_tensor_to_device(module, name, self.execution_device) File "/home/lab/user/anaconda3/envs/project/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 281,   
in set_module_tensor_to_device raise ValueError(f"{tensor_name} is on the meta device, we need a valueto put in on {device}.") ValueError: weight is on the meta device, we need avalue to put in on 6.

```

Importantly, loading the model with `AutoModelForCausalLM` instead of `AutoModelForSequenceClassification` works.  
It seems as if `AutoModelForSequenceClassification` creates the classification layers empty on the meta device, and then crushes when trying to move them to GPU.  
I tried loading everything directly into memory, without meta device, but couldn’t.

---

<div class="post-metadata">

**Author:** ![Glick](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@Glick](https://discuss.huggingface.co/u/Glick)\
**Post date:** [November 2, 2023, 9:54am UTC](https://discuss.huggingface.co/t/valueerror-weight-is-on-the-meta-device-when-using-auto-model-for-sequence-classification/58653/4 "2023-11-02T09:54:46Z")

</div>

A workaround I came with is loading the classification model to cpu and then saving it once using `accelerator.save_model`. Afterwards the model can be loaded using `load_checkpoint_and_dispatch`

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [November 30, 2023, 4:45pm UTC](https://discuss.huggingface.co/t/valueerror-weight-is-on-the-meta-device-when-using-auto-model-for-sequence-classification/58653/5 "2023-11-30T16:45:07Z")

</div>

Hi @Glick, could you try with this model instead `meta-llama/Llama-2-7b-hf` ?  
I’ve tested it on the latest transformers and I am unable to reproduce the issue.

```auto
from transformers import AutoModelForSequenceClassification
import torch
model = AutoModelForSequenceClassification.from_pretrained('meta-llama/Llama-2-7b-hf', 
                                                           torch_dtype=torch.float16, 
                                                           device_map='auto', 
                                                           num_labels=3)

```