# UnicodeDecodeError when loading Mulit Lingual text file

**URL:** <https://discuss.huggingface.co/t/unicodedecodeerror-when-loading-mulit-lingual-text-file/15698>\
**Category:** 🤗Datasets\
**Created:** [March 14, 2022, 4:31pm UTC](https://discuss.huggingface.co/t/unicodedecodeerror-when-loading-mulit-lingual-text-file/15698 "2022-03-14T16:31:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohamed-illiyas](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@mohamed-illiyas](https://discuss.huggingface.co/u/mohamed-illiyas)\
**Post date:** [March 14, 2022, 4:31pm UTC](https://discuss.huggingface.co/t/unicodedecodeerror-when-loading-mulit-lingual-text-file/15698/1 "2022-03-14T16:31:51Z")

</div>

I am loading a text file which contains various languages and emojis, characters, etc.  
Sample line in the text file: ज़ागान, पश्चिमी पोलैंड में एक शहर है। यह शहर बोबर नदी के किनारे स्थित है  
When loading the file with `load_dataset`, it is giving back some unicode decoding error.

Code I used:

```auto
from datasets import load_dataset
dataset = load_dataset('text', data_files="/Folder/path/to/file/file.txt")

```

Error:

```auto
Traceback (most recent call last):
  File "vocab.py", line 4, in <module>
    dataset = load_dataset('text', data_files="/Folder/path/to/file/file.txt")
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 683, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 1133, in _prepare_split
    for key, table in utils.tqdm(
  File "/opt/conda/lib/python3.8/site-packages/tqdm/std.py", line 1173, in __iter__
    for obj in iterable:
  File "/opt/conda/lib/python3.8/site-packages/datasets/packaged_modules/text/text.py", line 60, in _generate_tables
    batch = f.read(self.config.chunksize)
  File "/opt/conda/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data

```

How to avoid this error while loading text file containing multiple language characters ?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 7, 2022, 12:26pm UTC](https://discuss.huggingface.co/t/unicodedecodeerror-when-loading-mulit-lingual-text-file/15698/2 "2022-04-07T12:26:01Z")

</div>

Hi! You can find the file encoding with [this package](https://pypi.org/project/chardet/) and then pass it to `load_dataset` via the `encoding` parameter to successfully load the dataset.