# MLM train loss is very different after version update

**URL:** <https://discuss.huggingface.co/t/mlm-train-loss-is-very-different-after-version-update/9563>\
**Category:** 🤗Transformers\
**Created:** [August 29, 2021, 7:47am UTC](https://discuss.huggingface.co/t/mlm-train-loss-is-very-different-after-version-update/9563 "2021-08-29T07:47:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MrSky](https://avatars.discourse-cdn.com/v4/letter/m/7993a0/32.png) [@MrSky](https://discuss.huggingface.co/u/MrSky)\
**Post date:** [August 29, 2021, 7:47am UTC](https://discuss.huggingface.co/t/mlm-train-loss-is-very-different-after-version-update/9563/1 "2021-08-29T07:47:00Z")

</div>

Hi,  
I use the run\_mlm.py script to pretrain from scratch BERT model. Not sure if this is the most updated version of the script since I’ve been using it for a couple of months.  
When I used transofrmers **4.5.1** I used to get train loss of ~1.9 but after updating to transformers **4.9.2** the train loss is ~4.5.  
I’m training from scratch on my data file + trained tokenizer and I ran the exact same command on both times.

This are the training argument on 4.5.1:

output\_dir=test-mlm-wiki, overwrite\_output\_dir=True, do\_train=True, do\_eval=None, do\_predict=False, evaluation\_strategy=IntervalStrategy.NO, prediction\_loss\_only=False, per\_device\_train\_batch\_size=32, per\_device\_eval\_batch\_size=8, gradient\_accumulation\_steps=1, eval\_accumulation\_steps=None, learning\_rate=5e-05, weight\_decay=0.0, adam\_beta1=0.9, adam\_beta2=0.999, adam\_epsilon=1e-08, max\_grad\_norm=1.0, num\_train\_epochs=1.0, max\_steps=-1, lr\_scheduler\_type=SchedulerType.LINEAR, warmup\_ratio=0.1, warmup\_steps=0, logging\_dir=runs/Aug08\_11-51-18, logging\_strategy=IntervalStrategy.STEPS, logging\_first\_step=False, logging\_steps=500, save\_strategy=IntervalStrategy.STEPS, save\_steps=5000, save\_total\_limit=1, no\_cuda=False, seed=42, fp16=False, fp16\_opt\_level=O1, fp16\_backend=auto, fp16\_full\_eval=False, local\_rank=-1, tpu\_num\_cores=None, tpu\_metrics\_debug=False, debug=False, dataloader\_drop\_last=False, eval\_steps=500, dataloader\_num\_workers=0, past\_index=-1, run\_name=test-mlm-wiki, disable\_tqdm=True, remove\_unused\_columns=True, label\_names=None, load\_best\_model\_at\_end=False, metric\_for\_best\_model=None, greater\_is\_better=None, ignore\_data\_skip=False, sharded\_ddp=[], deepspeed=None, label\_smoothing\_factor=0.0, adafactor=False, group\_by\_length=False, length\_column\_name=length, report\_to=[‘tensorboard’], ddp\_find\_unused\_parameters=None, dataloader\_pin\_memory=True, skip\_memory\_metrics=False, \_n\_gpu=1, mp\_parameters=

These are the training arguments on 4.9.2:

TrainingArguments(\_n\_gpu=1, adafactor=False, adam\_beta1=0.9,  
adam\_beta2=0.999, adam\_epsilon=1e-08, dataloader\_drop\_last=False, dataloader\_num\_workers=0, dataloader\_pin\_memory=True, ddp\_find\_unused\_parameters=None, debug=[], deepspeed=None, disable\_tqdm=True, do\_eval=False, do\_predict=False, do\_train=True, eval\_accumulation\_steps=None, eval\_steps=None, evaluation\_strategy=IntervalStrategy.NO, fp16=False, fp16\_backend=auto, fp16\_full\_eval=False, fp16\_opt\_level=O1, gradient\_accumulation\_steps=1, greater\_is\_better=None, group\_by\_length=False, ignore\_data\_skip=False, label\_names=None, label\_smoothing\_factor=0.0, learning\_rate=5e-05, length\_column\_name=length, load\_best\_model\_at\_end=False, local\_rank=-1, log\_level=-1, log\_level\_replica=-1, log\_on\_each\_node=True, logging\_dir=test-mlm-wiki/runs/Aug27\_17-12-33, logging\_first\_step=False, logging\_steps=500, logging\_strategy=IntervalStrategy.STEPS, lr\_scheduler\_type=SchedulerType.LINEAR, max\_grad\_norm=1.0, max\_steps=-1, metric\_for\_best\_model=None, mp\_parameters=, no\_cuda=False, num\_train\_epochs=1.0, output\_dir=test-mlm-wiki, overwrite\_output\_dir=True, past\_index=-1, per\_device\_eval\_batch\_size=8, per\_device\_train\_batch\_size=32, prediction\_loss\_only=False, push\_to\_hub=False, push\_to\_hub\_model\_id=test-mlm-wiki-sst5-100, push\_to\_hub\_organization=None, push\_to\_hub\_token=None, remove\_unused\_columns=True, report\_to=[‘tensorboard’], resume\_from\_checkpoint=None, run\_name=test-mlm-wik, save\_on\_each\_node=False, save\_steps=5000, save\_strategy=IntervalStrategy.STEPS,  
save\_total\_limit=1, seed=42, sharded\_ddp=[], skip\_memory\_metrics=True, tpu\_metrics\_debug=False, tpu\_num\_cores=None, use\_legacy\_prediction\_loop=False, warmup\_ratio=0.1, warmup\_steps=0, weight\_decay=0.0)

These are the difference I’ve found:

| 4.5.1 | 4.9.2 |
| --- | --- |
| debug=False | debug=[] |
| eval\_steps=500 | eval\_steps=None |
| logging\_dir=runs/Aug08\_11-51-18\_dogfish-01 | logging\_dir=est-mlm-wiki/runs/Aug27\_17-12-33\_dogfish-01 |
| - | log\_level=-1 |
| - | log\_level\_replica=-1 |
| - | log\_on\_each\_node=True |
| - | push\_to\_hub=False |
| - | push\_to\_hub\_model\_id=test-mlm-wiki |
| - | push\_to\_hub\_organization=None |
| - | push\_to\_hub\_token=None |
| - | save\_on\_each\_node=False |
| - | use\_legacy\_prediction\_loop=False |
| - | resume\_from\_checkpoint=None |

Any ideas why this happens?

---

<div class="post-metadata">

**Author:** ![MrSky](https://avatars.discourse-cdn.com/v4/letter/m/7993a0/32.png) [@MrSky](https://discuss.huggingface.co/u/MrSky)\
**Post date:** [August 29, 2021, 11:10am UTC](https://discuss.huggingface.co/t/mlm-train-loss-is-very-different-after-version-update/9563/2 "2021-08-29T11:10:24Z")

</div>

I wonder if something is going on with the tokenizer…  
When running on the 4.5.1 version, I changed in run\_mlm script AutoTokenizer to BertTokenizer (it had a bug perhaps, wouldn’t recognize my tokenizer).  
On 4.9.2 I get this error when running the script:  
“The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.  
The tokenizer class you load from this checkpoint is ‘T5TokenizerFast’.  
The class this function is called from is ‘BertTokenizer’.”

That’s the code I’m using to train the tokenizer:  
tokenizer = BertWordPieceTokenizer()  
tokenizer.train(files=[‘wikipedia\_1e6.txt], vocab\_size=30000)  
tokenizer.save\_model(’/tokenizers/vocab\_folder’)  
tokenizer = BertTokenizer.from\_pretrained(’/tokenizers/vocab\_folder’)  
tokenizer.save\_pretrained(’/tokenizers/wikipedia\_1e6/’)

However, when training tokenizer with transformers 4.9.2 and using it in run\_mlm I still get a bigger train loss (than 4.5.1), but no warning.
