# Keeping only current dataset state in cache

**URL:** <https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740>\
**Category:** 🤗Datasets\
**Created:** [June 11, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740 "2021-06-11T20:02:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcelgwerder](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@marcelgwerder](https://discuss.huggingface.co/u/marcelgwerder)\
**Post date:** [June 11, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740/1 "2021-06-11T20:02:58Z")

</div>

Hi,

How would I apply multiple processing functions on a dataset while only ever keeping one version of the dataset in cache. The code below, for example, seems to keep every state of the dataset in the `/tmp` directory until the session ends. This results in my dataset being duplicated on disk many times, and thus requiring a ton of disk space.

```auto
set_caching_enabled(False)

dataset = load_dataset(...)

# Creates first copy in /tmp
dataset = dataset.map(...)
# Creates second copy in /tmp
dataset = dataset.map(...)
# Creates third copy in /tmp
dataset = dataset.map(...)

dataset.save_to_disk(...)

```

I don’t care about the intermediate versions. I’m running this as a SageMaker processing job and my whole container is discarded afterwards. All I need is an input and an output version of the dataset. How would I get rid of the additional copies in `/tmp` so I can reduce my scripts’ space requirements?

Thanks,  
Marcel

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 20, 2021, 10:27am UTC](https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740/2 "2021-07-20T10:27:33Z")

</div>

You can get the cache files to delete with `dataset.cache_files`. Feel free to delete them from your filesystem at each step:

```python
import os

def delete_dataset(dataset):
    cached_files = [cache_file["filename"] for cache_file in dataset.cache_files]
    del dataset
    for cached_file in cached_files:
        os.remove(cached_file)

# This line creates a new dataset using map, and deletes the old dataset
dataset, _ = dataset.map(...), delete_dataset(dataset)

```

---

<div class="post-metadata">

**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](https://discuss.huggingface.co/u/sriniv)\
**Post date:** [August 29, 2022, 11:45am UTC](https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740/3 "2022-08-29T11:45:26Z")

</div>

@lhoestq - If we delete cache files after each step, will it make the processing of data using .map() slower when compared to not deleting cache files?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 30, 2022, 9:57am UTC](https://discuss.huggingface.co/t/keeping-only-current-dataset-state-in-cache/6740/4 "2022-08-30T09:57:33Z")

</div>

No it won’t make it slower: it would just remove the intermediate steps that you don’t need anymore.

Though if you re-run your processing from scratch, it will recompute everything since the cache files were removed