# Missing keys in RobertaForMaskedLM state dict

**URL:** <https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279>\
**Category:** 🤗Transformers\
**Created:** [August 5, 2022, 1:45pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279 "2022-08-05T13:45:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](https://discuss.huggingface.co/u/adrienchaton)\
**Post date:** [August 5, 2022, 1:45pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/1 "2022-08-05T13:45:24Z")

</div>

Hello,

I am training my model that is based on RobertaForMaskedLM and after saving it I get a pytorch\_model.bin file from which I want to retrieve certain layer parameters.

After I do  
state\_dict = torch.load(path\_to\_pytorch\_model.bin, map\_location=‘cpu’)  
and inspect the state\_dict.keys() I observe that two parameters are missing: lm\_head.decoder.weight and lm\_head.decoder.bias

From what I understand, lm\_head.decoder.weight = roberta.embeddings.word\_embeddings.weight which is due to the tied weights.

What I have not figured out is where are stored the parameters of lm\_head.decoder.bias. Are these unused parameters ?

Thanks for your help !

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 5, 2022, 1:55pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/2 "2022-08-05T13:55:27Z")

</div>

Those are not used since they are tied weight. You should use `strict=False` when loading the model then retie the weights with `roberta.tie_weights()` (or jsut use `from_pretrained` which will do all of that for you).

---

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](https://discuss.huggingface.co/u/adrienchaton)\
**Post date:** [August 5, 2022, 2:20pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/3 "2022-08-05T14:20:18Z")

</div>

Thank you for your reply. These two ways you describe indeed give me same results  
calling from\_pretrained  
or load\_state\_dict with strict=False followed by tie\_weights()  
then the parameters for lm\_head.decoder are the same both in terms of weight (same as that of the embeddings.word\_embeddings) and bias.

What still puzzle me is to what is tied the bias parameters (or where it is stored because it is not in the state\_dict keys), this I could not get it from the source code.

If for example I set to zeros the lm\_head.decoder.bias then the model output is not the same anymore.  
And if I check for gradients during the backward pass, the param.grad from lm\_head.decoder.bias are neither None or zero which would mean it is also trained.

---

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](https://discuss.huggingface.co/u/adrienchaton)\
**Post date:** [August 5, 2022, 2:32pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/4 "2022-08-05T14:32:27Z")

</div>

going through all parameters with torch.allclose against lm\_head.decoder.weight and lm\_head.decoder.bias seems to have given me the answer

torch.allclose(roberta.embeddings.word\_embeddings.weight, lm\_head.decoder.weight) = True  
torch.allclose(lm\_head.bias, lm\_head.decoder.bias) = True

so it seems that lm\_head.decoder.bias and lm\_head.bias are tied … is that right ?  
do you know why is this bias parameter replicated ?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 5, 2022, 2:34pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/5 "2022-08-05T14:34:31Z")

</div>

This is a very hack way of ensuring `lm_head.bias` gets resized when we add new tokens, since only `lm_head.decoder` gets passed to that function, coupled with some technical debt of having created that weight in the first place instead of just relying on `lm_head.decoder`. Not the nicest part of the library 😅

---

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](https://discuss.huggingface.co/u/adrienchaton)\
**Post date:** [August 5, 2022, 2:38pm UTC](https://discuss.huggingface.co/t/missing-keys-in-robertaformaskedlm-state-dict/21279/6 "2022-08-05T14:38:13Z")

</div>

Ahah sorry for digging this out !  
But I needed to clarify that so thank you very much for helping.
