# IndexError using save\_to\_disk

**URL:** <https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911>\
**Category:** 🤗Datasets\
**Created:** [March 2, 2023, 5:41pm UTC](https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911 "2023-03-02T17:41:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bvandy](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@bvandy](https://discuss.huggingface.co/u/bvandy)\
**Post date:** [March 2, 2023, 5:41pm UTC](https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911/1 "2023-03-02T17:41:28Z")

</div>

In my notebook, I generate a python dictionary with data which is converted to a dataset using Dataset.from\_dict(). When I call save\_to\_disk on the dataset, I receive “IndexError: Index 1 out of range for dataset of size 1.” (see below for image of full error)

It is as though part of the datasets code thinks there is only 1 shard and another part of the datasets code thinks there should be 2 shards.

I found that if I specify num\_shards=1, then SOMETIMES the save\_to\_disk will work. If I try to specify max\_shard\_size, the IndexError is still seen.

When num\_shards=1 is set and the save does NOT work, there is no error message, the code window does finish processing, and the directory that should have the saved files is empty.

 ![numshards_fail](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a2141dc65ebe53018d50034bdacdc7dae73f5fbf.jpeg)

As seen in the screenshot, the dataset only contains a single row (which is correct), but it should be noted that the data for the ‘window\_incidices’ and ‘embeddings’ columns is very large. If num\_shards=1 is set and the dataset is saved to disk, the folder size is around 750 MB.

Python 3.8.10  
Datasets library 2.9.0

 ![indexerror](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e121c7cb5145cf9f1f620bea5972f202806474e6.jpeg)

---

<div class="post-metadata">

**Author:** ![bvandy](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@bvandy](https://discuss.huggingface.co/u/bvandy)\
**Post date:** [March 2, 2023, 6:24pm UTC](https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911/2 "2023-03-02T18:24:39Z")

</div>

After restarting the kernel and running it again, I was able to use num\_shards=1, but without num\_shards it still fails with the IndexError:

 ![indexerror2](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/c1d613229861714e90bcff1546de276ac32513c5.jpeg)

---

<div class="post-metadata">

**Author:** ![ProfessorQ](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@ProfessorQ](https://discuss.huggingface.co/u/ProfessorQ)\
**Post date:** [November 9, 2023, 10:42pm UTC](https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911/3 "2023-11-09T22:42:46Z")

</div>

Has this error been resolved?  
I noticed I have this error too.

---

<div class="post-metadata">

**Author:** ![lauriasuo](https://avatars.discourse-cdn.com/v4/letter/l/eada6e/32.png) [@lauriasuo](https://discuss.huggingface.co/u/lauriasuo)\
**Post date:** [February 1, 2024, 9:16am UTC](https://discuss.huggingface.co/t/indexerror-using-save-to-disk/32911/4 "2024-02-01T09:16:23Z")

</div>

Atleast in my case the index errors when saving a dataset with only one row were caused by the size of the row exceeding max\_shard\_size (default is 500MB).

When size of the dataset exceeds max\_shard\_size, the dataset is split into multiple shards. However, it seems that sharding is done row-wise: the writer expects at least as many rows as the number of shards. In the case of a dataset with one row whose size is larger than max\_shard\_size, this will cause an indexing error. I suspect the silent error might be caused by conflict from num\_shards=1 when size(row) \> max\_shard\_size.

Tldr; Either increase max\_shard\_size to be larger than the size of any individual row, or split your data into multiple rows.