# Training BERT model from scratch with custom sequence

**URL:** <https://discuss.huggingface.co/t/training-bert-model-from-scratch-with-custom-sequence/23462>\
**Category:** Beginners\
**Created:** [September 21, 2022, 10:37pm UTC](https://discuss.huggingface.co/t/training-bert-model-from-scratch-with-custom-sequence/23462 "2022-09-21T22:37:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![latent-observation](https://avatars.discourse-cdn.com/v4/letter/l/e9a140/32.png) [@latent-observation](https://discuss.huggingface.co/u/latent-observation)\
**Post date:** [September 21, 2022, 10:37pm UTC](https://discuss.huggingface.co/t/training-bert-model-from-scratch-with-custom-sequence/23462/1 "2022-09-21T22:37:50Z")

</div>

Hello,

I have a text file (separated by whitespace) containing sequences of strings (they are _not_ words, rather the data belongs to a different domain).  
I wanted to try pre-training a BERT model with this data.

I have seen tutorials with folks fine-tuning on different target datasets but I don’t think there is a official tutorial for pretraining with custom data. I think I need to index the data sequences first and then do the pre-training.

Are there any resources that could help me with this? Any pointers would be much appreciated.