# Saving dataset in the current state without cache **URL:** \ **Category:** šŸ¤—Datasets\ **Created:** [April 30, 2021, 8:03pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892 "2021-04-30T20:03:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![dropout05](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dropout05/32/5465_2.png) [@dropout05](https://discuss.huggingface.co/u/dropout05)\ **Post date:** [April 30, 2021, 8:03pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/1 "2021-04-30T20:03:39Z")
Hi! I am working with a wikipedia dataset and want to distribute it across my machines. It is preprocessed roughly like this ```auto TEST_SIZE = 10_000 VAL_SIZE = 3_000 wiki_processed = datasets.load_dataset("wikipedia", "20200501.en")["train"] wiki_processed = wiki_processed.map(map_fn, num_proc=20) wiki_processed = wiki_processed.filter(lambda x: bool(x["text"]), num_proc=20) dataset_dict_val = wiki_processed.train_test_split(test_size=VAL_SIZE, seed=42) dataset_dict = dataset_dict_val["train"].train_test_split(test_size=TEST_SIZE, seed=42) dataset_dict["validation"] = dataset_dict_val["test"] dataset_dict.save_to_disk("../data/wikipedia_rank") ``` Originally, wikipedia takes about 16Gb, but wikipedia\_rank is more than 90Gb. This happens, because every subset `train`, `validation`, `test` has the original `dataset.arrow` in its directory + cache files. ```auto ~/d/c/d/wikipedia_rank āÆāÆāÆ du -hs * 4.0K dataset_dict.json 16G test 61G train 16G validation ~/d/c/d/wikipedia_rank āÆāÆāÆ du -hs test/* 4.3M test/cache-0924ed04d225940e.arrow 4.1M test/cache-0ca033cdbd106a2d.arrow 4.3M test/cache-1a97b84558ed7b1d.arrow 4.3M test/cache-22a7c31dbee9c03c.arrow 80K test/cache-2bb6f88c5c8f4329.arrow 4.4M test/cache-2f128b0e055ed3cf.arrow 4.4M test/cache-3c97985cec531e1a.arrow 4.4M test/cache-482dc3ba3be09fbc.arrow 4.0M test/cache-4c932850df42ad39.arrow 4.7M test/cache-4ce30cf00c698ef6.arrow 4.7M test/cache-4e36968451cc71ba.arrow 4.5M test/cache-550a1d3d28c3487a.arrow 4.9M test/cache-599e1bb413fe3195.arrow 4.2M test/cache-637a2658f481ab57.arrow 4.7M test/cache-7c27537563c55dd1.arrow 5.1M test/cache-924c2f977e10633f.arrow 4.7M test/cache-9510e0cd305aa3fa.arrow 4.5M test/cache-9664e702137250f3.arrow 4.4M test/cache-9c3a9b369f101759.arrow 4.1M test/cache-ab0b7947030b6ce7.arrow 4.0M test/cache-e124027841f7ed5b.arrow 16G test/dataset.arrow # I want test/dataset.arrow only to hold 10K examples, not the whole wikipedia 200K test/dataset_info.json 4.0K test/state.json ``` What is the recommended way to only save a sampled and preprocessed version of my dataset without cache files? ---
**Author:** ![dropout05](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dropout05/32/5465_2.png) [@dropout05](https://discuss.huggingface.co/u/dropout05)\ **Post date:** [April 30, 2021, 8:44pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/2 "2021-04-30T20:44:58Z")
**UPD:** this workaround is not feasible in my case even with 1Tb of RAM =( Hereā€™s a workaround that I figured out, but it is slow and requires to put all of your data into RAM (and I donā€™t have enough). ```python dataset_dict = datasets.DatasetDict({ "train": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["train"].data.to_pydict()), "validation": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["validation"].data.to_pydict()), "test": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["test"].data.to_pydict()), }) dataset_dict.save_to_disk("../data/wikipedia_rank_nocache") ``` ---
**Author:** ![adilism](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@adilism](https://discuss.huggingface.co/u/adilism)\ **Post date:** [May 1, 2021, 9:21am UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/3 "2021-05-01T09:21:30Z")
Possibly [clearing cache](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.cleanup_cache_files) before saving would work? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [May 1, 2021, 3:36pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/4 "2021-05-01T15:36:42Z")
you can disable caching entirely by using: ```python import datasets datasets.set_caching_enabled(False) ``` ---
**Author:** ![dropout05](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dropout05/32/5465_2.png) [@dropout05](https://discuss.huggingface.co/u/dropout05)\ **Post date:** [May 3, 2021, 11:23pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/5 "2021-05-03T23:23:03Z")
Thank you [adilism](https://discuss.huggingface.co/u/adilism) and [lewtun](https://discuss.huggingface.co/u/lewtun), I tried both suggestions, but they did not work out for me šŸ˜ž. Neither `cleanup_cache_files()` nor `datasets.set_caching_enabled(False)` do not seem to affect saving at all, which is curious. I still have 16Gb for each train, validation, and test in my `wikipedia_rank_nocache` folders. The thing that kind of worked is to only create new `ArrowDataset` objects for validaiton and test. These two are small and fit into memory easily. ```auto dataset_dict["test"] = datasets.arrow_dataset.Dataset.from_dict(dataset_dict_val["test"].data.to_pydict()) dataset_dict["validation"] = datasets.arrow_dataset.Dataset.from_dict(dataset_dict_val["validation"].data.to_pydict()) dataset_dict.save_to_disk("../data/wikipedia_rank_nocache") ``` ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [May 5, 2021, 8:08am UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/6 "2021-05-05T08:08:46Z")
I had issues with caches as well, try calling flatten\_indices before saving the dataset. ---
**Author:** ![dropout05](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dropout05/32/5465_2.png) [@dropout05](https://discuss.huggingface.co/u/dropout05)\ **Post date:** [May 6, 2021, 7:01pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/7 "2021-05-06T19:01:12Z")
That worked! Thank you! ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [May 7, 2021, 9:40am UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/8 "2021-05-07T09:40:32Z")
You are welcome @dropout05 Also, @lhoestq please take note - I didnā€™t file this issue as it was a) transient across releases, b) it was minor, I found a workaround. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [May 7, 2021, 11:52am UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/9 "2021-05-07T11:52:30Z")
Indeed currently if you slice the dataset in some way (using `shard`, `train_test_split` or `select` for example), then under the hood the actual dataset isnā€™t changed, but instead an indices mapping is added to avoid having to rewrite a new arrow Table (save time + disk/memory usage). It maps the indices used by **getitem** to the right rows if the arrow Table. By default `save_to_disk` does save the full dataset table + the mapping. If you want to only save the shard of the dataset instead of the original arrow file + the indices, then you have to call `flatten_indices` first. It creates a new arrow table by using the right rows of the original table. The current documentation is missing this, let me update it. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 17, 2022, 5:57pm UTC](https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892/10 "2022-03-17T17:57:52Z")
Update, since this thread still has some views: Now `flatten_indices` is called before saving the dataset to disk by default, in order to avoid saving the full dataset if thereā€™s only a subset that needs to be written (after calling `select` or `shard` for example)