# Tutorial on Pretraining BERT

**URL:** <https://discuss.huggingface.co/t/tutorial-on-pretraining-bert/2828>\
**Category:** Beginners\
**Created:** [December 15, 2020, 8:58am UTC](https://discuss.huggingface.co/t/tutorial-on-pretraining-bert/2828 "2020-12-15T08:58:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![npan1990](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@npan1990](https://discuss.huggingface.co/u/npan1990)\
**Post date:** [December 15, 2020, 8:58am UTC](https://discuss.huggingface.co/t/tutorial-on-pretraining-bert/2828/1 "2020-12-15T08:58:10Z")

</div>

I would like to pretrain a BERT model with custom data. For example, data in my native language. Is there any example script about pretraining a BERT model? I found the tutorial with the EsperBERTo model but this is not a BERT model as far as I understand.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [December 15, 2020, 9:01am UTC](https://discuss.huggingface.co/t/tutorial-on-pretraining-bert/2828/2 "2020-12-15T09:01:11Z")

</div>

You can have a look at the example scripts here: [https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling)