# Newbie: Main difference between tokenizers?

**URL:** <https://discuss.huggingface.co/t/newbie-main-difference-between-tokenizers/6035>\
**Category:** 🤗Tokenizers\
**Created:** [May 6, 2021, 5:32pm UTC](https://discuss.huggingface.co/t/newbie-main-difference-between-tokenizers/6035 "2021-05-06T17:32:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![wusuowei60](https://avatars.discourse-cdn.com/v4/letter/w/edb3f5/32.png) [@wusuowei60](https://discuss.huggingface.co/u/wusuowei60)\
**Post date:** [May 6, 2021, 5:32pm UTC](https://discuss.huggingface.co/t/newbie-main-difference-between-tokenizers/6035/1 "2021-05-06T17:32:07Z")

</div>

Hi, HuggingFace community.

I’m new to NLP and have learned there are many packages that can do tokenization, such as spacy, NLTK, torchtext and of course HuggingFace tokenizer. My questions are:

1️⃣ What is the main difference between them?

It seems to me that 🤗 tokenizers are fast and match with corresponding models. Any difference other than that?

2️⃣ Why do different models need different tokenizers?

I heard that using wrong tokenizers may hurt performance, so tokenizers make a difference. Why is that the case, any intuition?

3️⃣ Which should I try first?

If I am to design my model (toy ones) instead of directly using provided API, which tokenizer should I try first.