# Cache is too large 300gb data -\> 3TB cache

**URL:** <https://discuss.huggingface.co/t/cache-is-too-large-300gb-data-3tb-cache/20603>\
**Category:** 🤗Datasets\
**Created:** [July 20, 2022, 3:50am UTC](https://discuss.huggingface.co/t/cache-is-too-large-300gb-data-3tb-cache/20603 "2022-07-20T03:50:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yiwc](https://avatars.discourse-cdn.com/v4/letter/y/c67d28/32.png) [@yiwc](https://discuss.huggingface.co/u/yiwc)\
**Post date:** [July 20, 2022, 3:50am UTC](https://discuss.huggingface.co/t/cache-is-too-large-300gb-data-3tb-cache/20603/1 "2022-07-20T03:50:18Z")

</div>

Dear Team,

I use the map to process data, then 300GB dataset becomes 3TB cache, and run out of my device storage.

possible solutions:

1. I understand we can use set\_transform to process on the fly. But how could I do remove\_columns after using set\_transfom? as I have a remove\_columns after map.  
 ![Screenshot 2022-07-20 at 11.49.50 AM](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0974eba95e80b5820e4d689966aff06328571df1.png)

May I know if you have a good solution for this? thank you!

---

<div class="post-metadata">

**Author:** ![yiwc](https://avatars.discourse-cdn.com/v4/letter/y/c67d28/32.png) [@yiwc](https://discuss.huggingface.co/u/yiwc)\
**Post date:** [July 21, 2022, 9:47am UTC](https://discuss.huggingface.co/t/cache-is-too-large-300gb-data-3tb-cache/20603/2 "2022-07-21T09:47:04Z")

</div>

solved with set\_transform, and remove columns within the preprocess\_function

---

<div class="post-metadata">

**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](https://discuss.huggingface.co/u/sriniv)\
**Post date:** [August 29, 2022, 11:35am UTC](https://discuss.huggingface.co/t/cache-is-too-large-300gb-data-3tb-cache/20603/3 "2022-08-29T11:35:19Z")

</div>

@yiwc - It would be of great help if you can add sample code on how you solved this.