# Working with large datasets **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 3, 2020, 4:43pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876 "2020-11-03T16:43:57Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 3, 2020, 4:43pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/1 "2020-11-03T16:43:57Z")
Hey @lhoestq, I am preparing datasets for BERT pre-training and often save\_to\_disk simply dies without saving the contents of the files to disk. The largest file I was able to save was 18 GB but above that, I am having no luck. Any performance tips for dealing with large datasets? Should I simply shard before saving to disk? If I do that, then I get copies of 18 GB files in each shard’s directory. What are my options? Forgot to mention. I am already using num\_proc and larger batches to speed up dataset map invocations. Those work great. It’s the save\_to\_disk that I am not sure how to deal with. And sharding without additional copies of the underlying dataset being copies in all shard directories. Thanks in advance. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 3, 2020, 6:22pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/2 "2020-11-03T18:22:32Z")
Hi ! Is there an error message ? ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 3, 2020, 6:39pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/3 "2020-11-03T18:39:09Z")
No, the Python process just dies. I have 120 GB of RAM on this machine and 500 GB of disk space. If I could shard this large dataset before saving that would work too. I would just save these shards one by one. But by doing sharding I get copies of of dataset.arrow in each dir and it quickly adds up. ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 4, 2020, 12:59pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/4 "2020-11-04T12:59:34Z")
I found a way to resolve the issue - shard the dataset before doing the last transformation and save\_to\_disk. That way resulting shards are not copies of dataset.arrow and indices file. If you are curious to have a look, here it is [create\_pretraining\_dataset.py](https://github.com/vblagoje/transformers/blob/new_trainer/examples/language-modeling/create_pretraining_dataset.py) ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 9, 2020, 1:36pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/5 "2020-11-09T13:36:32Z")
You had several copies of the full dataset file because the `shard` method only applies an indices mapping on top of the loaded dataset file: it doesn’t create a new dataset file. To remove the indices mapping and write one file per shard you have to call `flatten_indices` on each shard. Do you have a way to reproduce the crash you experienced ? ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 10, 2020, 9:54am UTC](https://discuss.huggingface.co/t/working-with-large-datasets/1876/6 "2020-11-10T09:54:53Z")
Gotcha, thanks! Perhaps `flatten_indices` could be a parameter when sharding? I didn’t see a crash any more as I moved onto this new approach. I am now trying to figure out how to create superfast dataloaders using datasets. If you got some tips on that one lmk.