# Difference between tokenizer and tokenizerfast

**URL:** <https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226>\
**Category:** Beginners\
**Created:** [May 18, 2021, 11:44am UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226 "2021-05-18T11:44:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ad26kr](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ad26kr](https://discuss.huggingface.co/u/ad26kr)\
**Post date:** [May 18, 2021, 11:44am UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226/1 "2021-05-18T11:44:34Z")

</div>

Hi,

I have searched for the answer for my question, but still can’t get the clear answer.

Some issues in the github/forum also report that the result of tokenizer and tokenizerfast is a little bit different.

I want to know what is the difference between them (in terms of mechanism)?  
If they should output the same result, then why we need both of them?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [May 18, 2021, 12:53pm UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226/2 "2021-05-18T12:53:04Z")

</div>

hey @ad26kr can you provide a few links on the reported differences between the two types of tokenizers?

cc @anthony who is the tokenizer expert

---

<div class="post-metadata">

**Author:** ![ad26kr](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@ad26kr](https://discuss.huggingface.co/u/ad26kr)\
**Post date:** [May 18, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226/3 "2021-05-18T13:12:39Z")

</div>

@anthony  
After careful reading of those posts, I found most of the different results from tokenizers/fast-tokenizers are solved.

However, I am still curious about the reason to separate them, since I found them very different in computation speed. Why not just use the fast one?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [May 18, 2021, 1:24pm UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226/4 "2021-05-18T13:24:03Z")

</div>

> [@ad26kr](#):
>
> However, I am still curious about the reason to separate them, since I found them very different in computation speed. Why not just use the fast one?

oh that’s because we do not have rust implementations + python bindings for every type of tokenizer that’s released by the various research groups. by default `transformers` will look for the fast implementation if it exists, or fall back to the “slow” one when it doesn’t

---

<div class="post-metadata">

**Author:** ![KhaiKit](https://avatars.discourse-cdn.com/v4/letter/k/3d9bf3/32.png) [@KhaiKit](https://discuss.huggingface.co/u/KhaiKit)\
**Post date:** [December 22, 2023, 3:27pm UTC](https://discuss.huggingface.co/t/difference-between-tokenizer-and-tokenizerfast/6226/5 "2023-12-22T15:27:03Z")

</div>

Hi @lewtun  
I was just wondering if the only difference between fast tokenizers and python tokenizers is really just speed?  
Reason being I saw that, for example for NLLB, the python tokenizer is based on SentencePiece while the fast tokenizer is based on BPE. Hence I was wondering if the output of the tokenizers is designed to be the same despite the difference in what it is based on?