# How to initialize a model with random weights

**URL:** <https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426>\
**Category:** Beginners\
**Created:** [October 22, 2024, 4:55pm UTC](https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426 "2024-10-22T16:55:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![GonRos22](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@GonRos22](https://discuss.huggingface.co/u/GonRos22)\
**Post date:** [October 22, 2024, 4:55pm UTC](https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426/1 "2024-10-22T16:55:44Z")

</div>

i’m debugging a script and since its very memory heavy my run gets stuck when i am evaluating in the console.

the only solution is to tweak some things and rerun the script. the problem is that for this i need to load the model which takes a few minutes every time.

is it possible to keep the configuration exactly as is but skip the model weights loading?

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [October 25, 2024, 12:11pm UTC](https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426/2 "2024-10-25T12:11:44Z")

</div>

The actual weights are loaded with the `from_pretrained()` class method. If you are only interested in the model’s skeleton, I’d do something like this:

```python
from transformers import AutoConfig, AutoModelForCausalLM

model_id = "gpt2" # or whatever model you plan to use
model = AutoModelForCausalLM.from_pretrained(model_id)
config = AutoConfig.from_pretrained(model_id)
print(type(model)) # then you see the model class: GPT2LMHeadModel

from transformers import GPT2LMHeadModel
model2 = GPT2LMHeadModel(config)

```

If you print out both architectures (just typing `model` or `model2`), you’ll see that they are exactly the same, the only difference being that the weights have been randomly initialized in the second case.

---

<div class="post-metadata">

**Author:** ![GonRos22](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@GonRos22](https://discuss.huggingface.co/u/GonRos22)\
**Post date:** [October 26, 2024, 12:07pm UTC](https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426/3 "2024-10-26T12:07:49Z")

</div>

it seems like this option does not retrieve the exact same config as from\_pretrained. here is the example with my code, some elements are taken from the Llava repo

```auto
from transformers import AutoConfig
config = AutoConfig.from_pretrained('liuhaotian/llava-v1.5-13b')
model = LlavaLlamaForCausalLM(config)

File "/llava/model/language_model/llava_llama.py", line 45, in __init__
    self.model = LlavaLlamaModel(config)
  File "/llava/model/language_model/llava_llama.py", line 38, in __init__
    super(LlavaLlamaModel, self). __init__ (config)
  File "/llava/model/llava_arch.py", line 32, in __init__
    super(LlavaMetaModel, self). __init__ (config)
  File "//lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 956, in __init__
    [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
  File "//lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 956, in <listcomp>
    [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
  File "//lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 756, in __init__
    self.self_attn = LLAMA_ATTENTION_CLASSES[config._attn_implementation](config=config, layer_idx=layer_idx)
  File "//lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 299, in __init__
    self.attention_dropout = config.attention_dropout
  File "//lib/python3.10/site-packages/transformers/configuration_utils.py", line 265, in __getattribute__
python-BaseException
    return super(). __getattribute__ (key)
AttributeError: 'LlavaConfig' object has no attribute 'attention_dropout'

```

the original from\_pretrained had additional arguments:

```auto
model = LlavaLlamaForCausalLM.from_pretrained(
                    model_args.model_name_or_path,
                    cache_dir=training_args.cache_dir,
                    attn_implementation=attn_implementation,
                    torch_dtype=(torch.bfloat16 if training_args.bf16 else None),
                    **bnb_model_from_pretrained_args
                )

```

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [October 28, 2024, 12:19pm UTC](https://discuss.huggingface.co/t/how-to-initialize-a-model-with-random-weights/113426/4 "2024-10-28T12:19:39Z")

</div>

Looks like you’re missing some details in that config… but it’s hard to tell without knowing what’s in `attn_implementation` or `bnb_model_from_pretrained_args`.

You can manually add to the config whatever you’re missing as follows:

```python
from llava.model.language_model.llava_llama import LlavaLlamaForCausalLM
from transformers import AutoConfig

config = AutoConfig.from_pretrained('liuhaotian/llava-v1.5-13b')
config.update({"attention_dropout": 0.1})

model = LlavaLlamaForCausalLM(config)

```

After doing this I don’t get your error anymore, but then it complains about a new attribute (“rope\_theta” this time). I guess all these missing attributes are in the additional arguments that you pass to the `from_pretrained()` method.