# Issue concatenating datasets

**URL:** <https://discuss.huggingface.co/t/issue-concatenating-datasets/28743>\
**Category:** 🤗Datasets\
**Created:** [January 1, 2023, 6:16pm UTC](https://discuss.huggingface.co/t/issue-concatenating-datasets/28743 "2023-01-01T18:16:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![PabloAMC](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@PabloAMC](https://discuss.huggingface.co/u/PabloAMC)\
**Post date:** [January 1, 2023, 6:16pm UTC](https://discuss.huggingface.co/t/issue-concatenating-datasets/28743/1 "2023-01-01T18:16:58Z")

</div>

I am trying to concatenate two datasets

```auto
from datasets import load_dataset, concatenate_datasets

movie = load_dataset("movie_rationales")
imdb = load_dataset("imdb")
imdb = imdb['train']

```

Then I adapt the movie dataset

```auto
movie_imdb_format = movie['train'].map(
    lambda x: {"text": x["review"]}
)
movie_imdb_format = movie_imdb_format.remove_columns(["review", "evidences"])

```

and aim to concatenate them

```auto
dataset_cc = concatenate_datasets([imdb, movie_imdb_format])

```

These both datasets output

```auto
Dataset({
    features: ['text', 'label'],
    num_rows: 25000
})
Dataset({
    features: ['label', 'text'],
    num_rows: 1600
})

```

However, I get an error

```auto
ValueError: The features can't be aligned because the key label of features {'label': ClassLabel(names=['NEG', 'POS'], id=None), 'text': Value(dtype='string', id=None)} has unexpected type - ClassLabel(names=['NEG', 'POS'], id=None) (expected either ClassLabel(names=['neg', 'pos'], id=None) or Value("null").

```

Any suggestion of why this may be happening and how to solve it?

---

<div class="post-metadata">

**Author:** ![PabloAMC](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@PabloAMC](https://discuss.huggingface.co/u/PabloAMC)\
**Post date:** [January 1, 2023, 11:18pm UTC](https://discuss.huggingface.co/t/issue-concatenating-datasets/28743/2 "2023-01-01T23:18:23Z")

</div>

In the end I have opted for

```auto
imdb_df = pd.DataFrame(imdb)
movie_df = pd.DataFrame(movie_imdb_format)
overall_df = pd.concat([imdb_df, movie_df])
dataset = Dataset.from_pandas(overall_df)

```

which solves the above problem. Though I am not sure why it was failing.

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [January 2, 2023, 2:12pm UTC](https://discuss.huggingface.co/t/issue-concatenating-datasets/28743/3 "2023-01-02T14:12:37Z")

</div>

The problem was that your datasets had different features order:

- imdb: `['text', 'label']`
- movie\_imdb\_format: `['label', 'text']`

You can fix this with one simple line of code:

```auto
movie_imdb_format = movie_imdb_format.cast(imdb.features)

```

Now that both datasets have the exact same format, the command `dataset_cc = concatenate_datasets([imdb, movie_imdb_format])` should work!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 3, 2023, 10:41am UTC](https://discuss.huggingface.co/t/issue-concatenating-datasets/28743/4 "2023-01-03T10:41:01Z")

</div>

Note that in recent versions of `datasets` it’s possible to concatenate two datasets with columns in different orders - in this case the columns are aligned automatically.