# Max\_step and generative dataset

**URL:** <https://discuss.huggingface.co/t/max-step-and-generative-dataset/11400>\
**Category:** Intermediate\
**Created:** [November 5, 2021, 3:43pm UTC](https://discuss.huggingface.co/t/max-step-and-generative-dataset/11400 "2021-11-05T15:43:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pkoiralap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pkoiralap/32/2358_2.png) [@pkoiralap](https://discuss.huggingface.co/u/pkoiralap)\
**Post date:** [November 5, 2021, 3:43pm UTC](https://discuss.huggingface.co/t/max-step-and-generative-dataset/11400/1 "2021-11-05T15:43:53Z")

</div>

I am fine tuning BERT for my domain using both MLM and NSP. I have used the `TextDatasetForNextSentencePrediction` for dataset along with `DataCollatorForLanguageModeling` for MLM.

The problem is with `TextDatasetForNextSentencePrediction` as it loads the entire data into the memory. So I tweaked it a bit and now it generates the data rather than storing everything in the memory.

However the problem is with the max\_step for training. Now that it is generative, the ` __len__ ` property can not be added to `TextDatasetForNextSentencePrediction` and I am getting an error regarding the same thing:

```auto
ValueError: train_dataset does not implement __len__ , max_steps has to be specified

```

Can I just specify an upper bound and it will figure it out automatically? Or will it break? Because if it does, I can’t run multiple epochs. (right)?

There are a lot of X, Y questions here and I can see this question is rather open ended. But any sort of advice or direction is much appreciated.

Thanks