# UTF-16 for datasets?

**URL:** <https://discuss.huggingface.co/t/utf-16-for-datasets/43828>\
**Category:** 🤗Datasets\
**Created:** [June 19, 2023, 6:13pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828 "2023-06-19T18:13:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mtmahe](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mtmahe](https://discuss.huggingface.co/u/mtmahe)\
**Post date:** [June 19, 2023, 6:13pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828/1 "2023-06-19T18:13:00Z")

</div>

Working on a translator, hoping to do fine-tuning with a utf-16 dataset so I can get all the French accents etc.

Datasets load\_dataset() doesn’t seem to like non-utf-8

Is there a way to specify or does it HAVE to be utf-8?

If it has to be utf-8, any suggestions for special characters?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 19, 2023, 7:16pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828/2 "2023-06-19T19:16:17Z")

</div>

You can pass `encoding="utf-16"` to the `load_dataset` call.

---

<div class="post-metadata">

**Author:** ![mtmahe](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mtmahe](https://discuss.huggingface.co/u/mtmahe)\
**Post date:** [June 19, 2023, 7:28pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828/3 "2023-06-19T19:28:43Z")

</div>

“got an unexpected keyword argument ‘encoding’”

datasets==2.3.2

my line looks like this  
raw\_dataset = load\_dataset(“json”, data\_files=“./train.json”, encoding=“utf-16”)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 21, 2023, 1:34pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828/4 "2023-06-21T13:34:36Z")

</div>

I’ve merged a PR that adds this param to the JSON builder.

You can access this feature by installing `datasets` from `main` with `pip install git+https://github.com/huggingface/datasets.git`.

---

<div class="post-metadata">

**Author:** ![mtmahe](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mtmahe](https://discuss.huggingface.co/u/mtmahe)\
**Post date:** [June 21, 2023, 2:21pm UTC](https://discuss.huggingface.co/t/utf-16-for-datasets/43828/5 "2023-06-21T14:21:20Z")

</div>

Thanks Mario! You sir are a gentleman and a scholar. 😉