# Turkish characters gets corrupted when loading dataset via audiofolder **URL:** \ **Category:** 🤗Datasets\ **Created:** [April 3, 2023, 8:57am UTC](https://discuss.huggingface.co/t/turkish-characters-gets-corrupted-when-loading-dataset-via-audiofolder/35390 "2023-04-03T08:57:37Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![yilmazay](https://avatars.discourse-cdn.com/v4/letter/y/bbce88/32.png) [@yilmazay](https://discuss.huggingface.co/u/yilmazay)\ **Post date:** [April 3, 2023, 8:57am UTC](https://discuss.huggingface.co/t/turkish-characters-gets-corrupted-when-loading-dataset-via-audiofolder/35390/1 "2023-04-03T08:57:37Z")
Hi Everyone, I tried an xlsr finetuning using CommonVoice Turkish data. It get trained successfully and I can decode Turkish waves although the accuracy not so good. The decoded Turkish strings are valid, without any non-Turkish characters. Then, I fine-tuned xlsr again with our own Turkish data in the same way as described in a blog. Although finetuning completed without errors and I can decode Turkish audios with that generated model, however, the decoded text is very strange. There are a lot of diacritic dots almost on all characters. In Turkish no diacritic dots are used, so the decoded text looks very weird and anormal (as the following text): _ **ė ̇ȯṅu̇ṅ ̇bu̇ ̇ṡë̇v̇ṁî̇rî̇â̇i̇ṅ ̇ȯṙṫȧḋȧn ̇k̇ȧl̇düṙü̇yi̇ṁḋėṙk̇ė ̇k̇ȧrȧḋȧ ̇k̇ȧl̇ḋü̇ṙmü̇ğ̇ ̇î̇l̇k̇yėṫtė ̇kȧl̇düṙṁü̇ğ̇ ̇i̇ṅṡȧṅ ̇** _ When I look at the loaded data which are saved as arrow files in the cache, they are also corrupted. (as seen below): _ **“her ne hikmetse tekrar canlı yayın sesi aynı ÅŸekilde devam ediyoristemiyorum tabii böyle bir ÅŸey söz konusu deÄŸil böyle bir uygulama”** _. In addition, arrow files’ file encoding becomes ASCII. ( I was expecting them to be UTF-8) Whereas my input metadata.csv is totally OK and encoded as UTF-8. Thus, I’ve come up to the idea that the data gets corrupted while loading the dataset data. So, it seem to me that load\_dataset() function has problems with UTF-8 encoded data. I m loading data as below saibld\_train = load\_dataset(“audiofolder”, data\_dir=train\_data\_path) where train\_data\_path is a local folder in which a UTF-8 encoded metadata.csv and the audio files reside. My question is: What should I do to force load\_data() function of datasets to be able to correctly load non ASCII, UTF-8 (Turkish characters) without getting corrupted? Any help or recommendation is very much appreciated. Thanks in advance. Yılmaz A. ---
**Author:** ![yilmazay](https://avatars.discourse-cdn.com/v4/letter/y/bbce88/32.png) [@yilmazay](https://discuss.huggingface.co/u/yilmazay)\ **Post date:** [April 4, 2023, 8:38am UTC](https://discuss.huggingface.co/t/turkish-characters-gets-corrupted-when-loading-dataset-via-audiofolder/35390/2 "2023-04-04T08:38:32Z")
After a lot of debug and analysis, I found the root cause of the problem. It happened to be the input metadata.csv file’s encoding problem. It was not load\_dataset()'s fault. While I am generating csv, I convert all content into lowercase using Turkish locale. Interestingly Turkish upper case I, which is İ, uppercase I with a dot on top of it, was causing the problem. While converting İ to i, it is adding a diacritic dot on the next character. That was really very strange Any ways, I think the problem will be solved after a new training from scratch.