# Saving train/val/test datasets

**URL:** <https://discuss.huggingface.co/t/saving-train-val-test-datasets/9424>\
**Category:** 🤗Datasets\
**Created:** [August 23, 2021, 10:37pm UTC](https://discuss.huggingface.co/t/saving-train-val-test-datasets/9424 "2021-08-23T22:37:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [August 23, 2021, 10:37pm UTC](https://discuss.huggingface.co/t/saving-train-val-test-datasets/9424/1 "2021-08-23T22:37:27Z")

</div>

Hi everyone.

After creating a dataset consisting of all my data, I split it in train/validation/test sets. Following that, I am performing a number of preprocessing steps on all of them, and end up with three altered datasets, of type `datasets.arrow_dataset.Dataset`.

In order to save them and in the future load directly the preprocessed datasets, would I have to call

`dataset.save_to_disk(FILE_PATH)`

3 times, one for the training, one for the validation and one for the test set? Or is there any way to somehow save them all together? If yes, what is more efficient?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 25, 2021, 10:35am UTC](https://discuss.huggingface.co/t/saving-train-val-test-datasets/9424/2 "2021-08-25T10:35:44Z")

</div>

Hi !

You can save them all as a dataset dictionary:

```python
from datasets import DatasetDict, load_from_disk

dataset = DatasetDict({
    "train": train_dataset,
    "validation": validation_dataset,
    "test": test_dataset,
})

dataset.save_to_disk("path/to/dataset/dir")

# reload
dataset = load_from_disk("path/to/dataset/dir")

# access any split
train_dataset = dataset["train"]

```

This is especially useful to save several splits of a dataset together.

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [August 25, 2021, 7:09pm UTC](https://discuss.huggingface.co/t/saving-train-val-test-datasets/9424/3 "2021-08-25T19:09:03Z")

</div>

This is exactly what I was looking for!

Thanks a lot 🙂