# Batching vs. Sharding a Large Dataset

**URL:** <https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545>\
**Category:** 🤗Datasets\
**Created:** [June 3, 2021, 9:35am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545 "2021-06-03T09:35:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tadf](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@tadf](https://discuss.huggingface.co/u/tadf)\
**Post date:** [June 3, 2021, 9:35am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545/1 "2021-06-03T09:35:03Z")

</div>

Hi,  
I have a very large dataset that fails in the preprocess stage due to its size.  
I’ve looked into: sharding the dataset vs. using batched=True on the map(…) function.  
What are the advantages/use cases of either cases?

Also can anyone point me to an example that uses each methodology?  
Shards - The official docs lacks the use of shards (eg. how to do a full train using shards – do you run the training scheme such that you do 1 epoch for the shard and then switch to the next shard?)

Batched - Not sure what it means by:  
`Provided function which is applied to all elements of table returns a dict of types [<class 'list'>, <class 'list'>, <class 'str'>]. When using batched=True, make sure provided function returns a dict of types like (<class 'list'>, <class 'numpy.ndarray'>)`  
Any clear example showing `batched=True` will help!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 4, 2021, 9:34am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545/2 "2021-06-04T09:34:48Z")

</div>

Hi ! Do you need to shard your dataset because you don’t have enough disk space ?  
Have you considered doing the preprocessing on-the-fly during training to avoid filling up your disk ?

---

<div class="post-metadata">

**Author:** ![tadf](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@tadf](https://discuss.huggingface.co/u/tadf)\
**Post date:** [June 7, 2021, 1:42am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545/3 "2021-06-07T01:42:58Z")

</div>

Hi!  
I have 3TB of space, 128GB of memory. The custom dataset I’m using is audio + transcript(text) data which is about 100GB.  
By preprocessing on-the-fly, are you referring to `set_transform`?

---

<div class="post-metadata">

**Author:** ![tadf](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@tadf](https://discuss.huggingface.co/u/tadf)\
**Post date:** [June 8, 2021, 7:11am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545/5 "2021-06-08T07:11:30Z")

</div>

@lhoestq  
I was able to resolve the issue with `set_transform` along with `remove_unused_columns=False` on the Trainer!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 8, 2021, 11:09am UTC](https://discuss.huggingface.co/t/batching-vs-sharding-a-large-dataset/6545/6 "2021-06-08T11:09:13Z")

</div>

Hi ! I’m glad you managed to resolve your issue 🙂

Indeed to use the `Trainer` with a dataset that uses a transform, you must set remove\_unused\_columns to False