# Tokenizer vs. TokenizerFast

**URL:** <https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187>\
**Category:** 🤗Transformers\
**Created:** [August 12, 2021, 4:14pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187 "2021-08-12T16:14:20Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cattana](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@cattana](https://discuss.huggingface.co/u/cattana)\
**Post date:** [August 12, 2021, 4:14pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/1 "2021-08-12T16:14:20Z")

</div>

Hi,

When adding a new token in the vocabulary, there is a difference between `Tokenizer` and `FastTokenizer`.

```auto
from transformers import BartTokenizer, BartTokenizerFast

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large') 
tokenizer_fast = BartTokenizerFast.from_pretrained('facebook/bart-large') 

tokenizer.add_tokens("<NEW_TOKEN>")
tokenizer_fast.add_tokens("<NEW_TOKEN>")

sentence = "I added a <NEW_TOKEN> in the vocabulary."

print(tokenizer.encode(sentence)) 
# [0, 100, 355, 10, 50265, 179, 5, 32644, 4, 2]

print(tokenizer_fast.encode(sentence))
# [0, 100, 355, 10, 1437, 50265, 11, 5, 32644, 4, 2]

```

The fast tokenizer adds a space token before the \<NEW\_TOKEN\> (1437) while the standard tokenizer removes the automatic space from the next token (`179` vs. `11`).

I tried with RoBERTa and got the same problem.

Thanks!

---

<div class="post-metadata">

**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\
**Post date:** [August 12, 2021, 4:20pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/2 "2021-08-12T16:20:50Z")

</div>

Technically speaking overall implementation of tokenizers wrt to Sentencepiece is kind of hacky in HuggingFace.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 12, 2021, 4:54pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/3 "2021-08-12T16:54:07Z")

</div>

To control whether or not the space is added with fast tokenizers, you need to wrap it in an `AddedToken`:

```auto
from transformers import AddedToken

tokenizer_fast.add_tokens(AddedToken("<NEW_TOKEN>", lstrip=True))

```

You can also choose if you want to remove or not the space after with the `rstrip` argument.

@s4sarath Your remark is completely out of order, especially since BART uses a byte-level BPE tokenizer.

---

<div class="post-metadata">

**Author:** ![cattana](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@cattana](https://discuss.huggingface.co/u/cattana)\
**Post date:** [August 12, 2021, 5:18pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/4 "2021-08-12T17:18:18Z")

</div>

> [@sgugger](#):
>
> `trip=True`

Thanks!

So it works well for the left space. However, keeping the right space using the `rstrip` argument doesn’t work properly in the standard tokenizer.

```auto
token = AddedToken("<NEW_TOKEN>", lstrip=True, rstrip=False)
tokenizer.add_tokens(token)
tokenizer_fast.add_tokens(token)

print(tokenizer.encode(sentence)) 
# [0, 100, 355, 10, 50265, 179, 5, 32644, 4, 2]

print(tokenizer_fast.encode(sentence))
#[0, 100, 355, 10, 50265, 11, 5, 32644, 4, 2]

```

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 12, 2021, 5:55pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/5 "2021-08-12T17:55:05Z")

</div>

Yes, Python tokenizer do not use the `AddedToken` type, you should use the fast tokenizer whenever available as it has more functionality.

---

<div class="post-metadata">

**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\
**Post date:** [August 12, 2021, 9:56pm UTC](https://discuss.huggingface.co/t/tokenizer-vs-tokenizerfast/9187/6 "2021-08-12T21:56:51Z")

</div>

@sgugger - I agree. That’s why I specifically mentioned wrt to Sentencepiece.