# I need help to run my code on mlm task

**URL:** <https://discuss.huggingface.co/t/i-need-help-to-run-my-code-on-mlm-task/11944>\
**Category:** Beginners\
**Created:** [November 17, 2021, 7:43am UTC](https://discuss.huggingface.co/t/i-need-help-to-run-my-code-on-mlm-task/11944 "2021-11-17T07:43:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\
**Post date:** [November 17, 2021, 7:43am UTC](https://discuss.huggingface.co/t/i-need-help-to-run-my-code-on-mlm-task/11944/1 "2021-11-17T07:43:55Z")

</div>

I am trying to train T5 with two layers on both decoder and decoder(2 layers encoder, 2layers decoder) on MLM task, I have already built my own model based on [transformers.T5ForConditionalGeneration](https://huggingface.co/transformers/model_doc/t5.html#transformers.T5ForConditionalGeneration) that is why I need to train this version on MLM not using jax model as [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)  
that is why I tried to memic the code but using a pure torch and this is the link to my code. Ihave many questions:

- using t5-small tokenizer I go Error(/opt/conda/conda-bld/pytorch\_1634272204863/work/aten/src/ATen/native/cuda/Indexing.cu:699: indexSelectLargeIndex: block: [117,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.  
)

- I understand the difference between the pre-rained T5 models is the number of layers and consequently the number of parameters. But what is the difference then between the pre-trained tokenizers? I mean All models are pre-trained on c4, if the tokenizer is also trained on c4 corpus then why load the tokenizer with different names? Is the pre-trained tokenizer is the same for all models but when loading the pre-trained tokenizer we refer to the config of the pre-trained model which inside has the path to the same pre-trained tokenizer?  
Actually, I have tried the three tokenizers (small, base, big) to tokenize small samples of texts I did not notice any difference. Comparing the vocabulary of the three tokenizers I found that it is the same vocab for all tokenizers.  
Another question please and correct me if I am wrong. As to my knowledge, the tokenizer and data distribution go in parallel to train any model. If I want to pre-train the T5 model for different numbers of layers on Masked language modeling on (let us say any English text dataset from hugging face). Do I need to train the tokenizer for this corpus or it is enough to use pre trained T5 tokenizer?

- I tried to train the tokenizer as [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer-2) on wikitext train split(code in src folder) i do not have the error of indexing as above, but it freezes after lunching and I do not know why since nothing is written. You can find my code `[here](https://drive.google.com/drive/folders/1OiuQ_-hcEWDzUxjcC3PoRy4Hy0pYKtt2?usp=sharing)

Also using different tokenizers I got different numbers of examples to train(maybe cause the number of resulted tokens differs according to the tokenizer)