# Does training tokenizer and adding new token to model when training BART on custom dataset improve performance?

**URL:** <https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163>\
**Category:** Beginners\
**Created:** [September 16, 2020, 12:36pm UTC](https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163 "2020-09-16T12:36:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](https://discuss.huggingface.co/u/LiuYangyang)\
**Post date:** [September 16, 2020, 12:36pm UTC](https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163/1 "2020-09-16T12:36:11Z")

</div>

I have a dataset composed of 300,000 articles. Is it wise to train tokenizer on my own dataset and add new token to the model then train BART?  
Is my dataset large enough to pre-train the embedding of new tokens added?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 16, 2020, 1:24pm UTC](https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163/2 "2020-09-16T13:24:50Z")

</div>

If the domain of your dataset isn’t very specialized from the pre-training datatset then training tokenizer won’t help much. Also note that if you train a new tokenizer then you’ll need to do the pre-training again, and when you train your tokenizer from scratch you don’t need to add the new tokens any more since you already trained tokenizer. Hope this makes sense.

> Is my dataset large enough

IMO this is highly subjective question and depends on the quality of the data, the task etc.  
@sshleifer might have some advice for this

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 16, 2020, 4:42pm UTC](https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163/3 "2020-09-16T16:42:10Z")

</div>

Interested to see if this improves performance, but I suspect it will not.

---

<div class="post-metadata">

**Author:** ![surya-narayanan](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@surya-narayanan](https://discuss.huggingface.co/u/surya-narayanan)\
**Post date:** [May 1, 2023, 7:14pm UTC](https://discuss.huggingface.co/t/does-training-tokenizer-and-adding-new-token-to-model-when-training-bart-on-custom-dataset-improve-performance/1163/4 "2023-05-01T19:14:28Z")

</div>

Hi,

I’m just curious, on this thread- is it possible to make a model tokenizer agnostic? It seems that for every new domain, one would have to create a tokenizer and then retrain the model, which seems highly inefficient.