# Error with \<|endoftext|\> in Tokenizer GPT2

**URL:** <https://discuss.huggingface.co/t/error-with-endoftext-in-tokenizer-gpt2/2838>\
**Category:** 🤗Tokenizers\
**Created:** [December 15, 2020, 9:34pm UTC](https://discuss.huggingface.co/t/error-with-endoftext-in-tokenizer-gpt2/2838 "2020-12-15T21:34:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pivasil](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@pivasil](https://discuss.huggingface.co/u/pivasil)\
**Post date:** [December 15, 2020, 9:34pm UTC](https://discuss.huggingface.co/t/error-with-endoftext-in-tokenizer-gpt2/2838/1 "2020-12-15T21:34:00Z")

</div>

Hi!  
I work with the [sberbank-ai/rugpt3large\_based\_on\_gpt2](https://huggingface.co/sberbank-ai/rugpt3large_based_on_gpt2?text=) . It is the model for the russian language corpus.  
I need to implement the function:

```auto
def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)

```

from ( [#473](https://github.com/huggingface/transformers/issues/473))  
For this function to work correctly with a single token, I need to add a special token \<|endoftext|\>.  
But when I pass the string “\<|endoftext|\>token” to Toktokenizer, an error is returned “ValueError: type of None unknown: \<class ‘NoneType’\>. Should be one of a python, numpy, pytorch or tensorflow object”.  
But even if there are multiple tokens at the input of tokenizer, there will be this error too.  
Token \<|endoftext|\> is absend in the tokenizer dictionary. But in the dictionary of special tokens, this token is present.  
My questions:

1. What am I doing wrong?
2. How can I solve this problem?
3. Why is the \<|endoftext|\> absent in the dictionary?

This is short test shows my problem:

```auto
    #!pip install transformers
    from transformers import GPT2Tokenizer
    from transformers import GPT2LMHeadModel
    with torch.no_grad():
        #view of special tokens in dicronary
      print('#' * 20, ' view of special tokens in dictonary ', '#' * 20)
      items = tokenizer.get_vocab().items()
      for item in items:
        if (item[0].startswith('<')==True) and (item[0].endswith('>')==True):
          print(item)
      #view of special tokens
      print('#' * 20, 'map of special_tokens ', '#' * 20)
      print(tokenizer.special_tokens_map)
      #try to get the id's with <|endoftext|>
      print('#' * 20, " try to get the id's with <|endoftext|> ", '#' * 20,)
      single_token = 'вот'
      single_token_with_eos = '<|endoftext|>' + single_token
    #error here!
      id = tokenizer.encode(single_token_with_eos, return_tensors='pt') print('single_token_with_eos_id',tokenizer.encode(single_token_with_eos))

Output:
#################### view of special tokens in dictonary ####################
('<pad>', 0)
('<s>', 1)
('</s>', 2)
('<unk>', 3)
('<mask>', 4)
> ################# map of special_tokens ####################
> {'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>'}
> #################### try to get the id's with <|endoftext|> ####################
Error here!
...
```

---

<div class="post-metadata">

**Author:** ![pivasil](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@pivasil](https://discuss.huggingface.co/u/pivasil)\
**Post date:** [December 16, 2020, 6:59am UTC](https://discuss.huggingface.co/t/error-with-endoftext-in-tokenizer-gpt2/2838/2 "2020-12-16T06:59:46Z")

</div>

Sorry, I realized my mistake. It should have been like this:  
tokenizer = AutoTokenizer.from\_pretrained…  
model = AutoModelWithLMHead.from\_pretraine…

 ![Снимок экрана 2020-12-16 095902](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e552e3d5e3b0a53ddf5a9b47d19ea86babd22dbe.jpeg)  
“If nothing works, then read the instructions”

---

<div class="post-metadata">

**Author:** ![pivasil](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@pivasil](https://discuss.huggingface.co/u/pivasil)\
**Post date:** [December 16, 2020, 9:30am UTC](https://discuss.huggingface.co/t/error-with-endoftext-in-tokenizer-gpt2/2838/3 "2020-12-16T09:30:18Z")

</div>

No, this is also not correct. \< | endoftext / \> is missing in this model. See [https://github.com/sberbank-ai/ru-gpts/issues/28](https://github.com/sberbank-ai/ru-gpts/issues/28)