# Which encoding does GPT2 vocabulary file use?

**URL:** <https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875>\
**Category:** Beginners\
**Created:** [July 31, 2021, 4:28pm UTC](https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875 "2021-07-31T16:28:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArtemS](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ArtemS](https://discuss.huggingface.co/u/ArtemS)\
**Post date:** [July 31, 2021, 4:28pm UTC](https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875/1 "2021-07-31T16:28:12Z")

</div>

I wanted to investigate a little, how tokenizer works. But the model trained in Russian language (mostly), and I see garbage instead of tokens. I tried obvious variants like open in UTF-8, that didn’t help.

---

<div class="post-metadata">

**Author:** ![ArtemS](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ArtemS](https://discuss.huggingface.co/u/ArtemS)\
**Post date:** [July 31, 2021, 7:45pm UTC](https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875/2 "2021-07-31T19:45:31Z")

</div>

Well, I checked that json vocabulary loads as gibberish:

```auto
import json

with open("/content/notebooks/ru-gpts/models/gpt3large/vocab.json", "r", encoding="utf-8") as f:

  vocab = json.load(f)

list(vocab.items())[1000:1010]

```

```auto
[('Ñĥма', 1000),
 ('Ġпи', 1001),
 ('Ġn', 1002),
 ('ĠнеÑĤ', 1003),
 ('иÑĤа', 1004),
 ('ÑĢÑĥп', 1005),
 ('ec', 1006),
 ('енÑĭ', 1007),
 ('ĠÑıв', 1008),
 ('Ðĵ', 1009)]

```

The same exact gibberish is in tokenizer itself, which I view with `list(tok1.get_vocab().items())[1000:1010]`

Yet, the model seems to work somehow, and produce meaningful results in Russian.

---

<div class="post-metadata">

**Author:** ![ArtemS](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ArtemS](https://discuss.huggingface.co/u/ArtemS)\
**Post date:** [July 31, 2021, 8:29pm UTC](https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875/3 "2021-07-31T20:29:57Z")

</div>

Looks like utf8 saved as cp1252 to me.

```auto
test = u'ĠоÑĤвеÑĩаеÑĤ'

test.encode('cp1252', errors='replace').decode('utf8', errors='replace')

```

`?о�?ве�?ае�?`

Not sure what to do, since obviously some letters just destroyed with that conversion.

---

<div class="post-metadata">

**Author:** ![ArtemS](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ArtemS](https://discuss.huggingface.co/u/ArtemS)\
**Post date:** [August 1, 2021, 5:07pm UTC](https://discuss.huggingface.co/t/which-encoding-does-gpt2-vocabulary-file-use/8875/4 "2021-08-01T17:07:40Z")

</div>

Well, I figured out that there is more than just “latin-1”\<-\>“utf-8” back and forth.

There are some reasons why GPT2 changes some Unicode numbers, for example: [Why \u0120 (Ġ) is in so many pairs? · Issue #80 · openai/gpt-2 · GitHub](https://github.com/openai/gpt-2/issues/80)

Some Russian letters were shifted too for some reason. I guess the best way to decode them to something meaningful is like this:

```auto
rus = []
for token in tok1.get_vocab().keys():
  rus.append(tok.convert_tokens_to_string(token))
rus

```

```auto
' даже',
 'зы',
 'вал',
 'стро',
 ' очень',
 ' ник',
 ' р',
 ' можно',
 ' произ',
 'еле',
 'руд',

```