# Tokenizer mapping the same token to multiple token\_ids

**URL:** <https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328>\
**Category:** 🤗Tokenizers\
**Created:** [April 18, 2024, 2:56am UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328 "2024-04-18T02:56:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chahnwoo](https://avatars.discourse-cdn.com/v4/letter/c/a88e57/32.png) [@Chahnwoo](https://discuss.huggingface.co/u/Chahnwoo)\
**Post date:** [April 18, 2024, 2:56am UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328/1 "2024-04-18T02:56:35Z")

</div>

Why do tokenizers sometimes map the same text to different tokens?

```auto
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('TeamUNIVA/Komodo_7B_v1.0.0')

short_text = "tokenize" 
long_text = "tokenizer.tokenize() vs tokenizer()"
encoded_short = tokenizer.encode(short_text, add_special_tokens=False)
encoded_long = tokenizer.encode(long_text, add_special_tokens=False)

print("===== SHORT TEXT =====")
for token_id in encoded_short:
    print(f"{str(token_id)} : '{tokenizer.decode(token_id)}'")

print("===== LONG TEXT =====")
for token_id in encoded_long:
    print(f"{str(token_id)} : '{tokenizer.decode(token_id)}'")

```

If you run the code I’ve provided above, you’ll see the following result:

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/1/b175efd12de91662a6ae18775870653e112d91b4.png)

What confuses me about this result is the fact that the same word (“tokenize” in this case), is tokenized differently at different points of the text. Specifically, the token “token” seems to have two corresponding token ids.

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/1/31e7d7029df6c149949e56b4ed516f27c74c2a5e.png)

Here are some questions I have regarding the above:

1. Is it normal that the same token is mapped to multiple token\_ids?
2. In the case that it IS normal, is there any way to know which token\_id the token will be mapped to without having to encode the text?
3. In the case that it IS NOT normal, would you recommend that I use a different model?

The problem is that I’m trying to create “labels” for a particular dataset by masking up to a specific index of the “input\_ids” returned by the tokenizer. The code I am working on right now determines that specific index by searching for a specific sequence of tokens (the token\_ids that correspond to a specific part of my input prompt). If the tokenizer I am using is not consistent in the way that it encodes the same tokens, it would probably lead to failures with this particular approach.

I would appreciate any guidance, and thank you for reading!

---

<div class="post-metadata">

**Author:** ![Chahnwoo](https://avatars.discourse-cdn.com/v4/letter/c/a88e57/32.png) [@Chahnwoo](https://discuss.huggingface.co/u/Chahnwoo)\
**Post date:** [April 22, 2024, 12:22am UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328/2 "2024-04-22T00:22:51Z")

</div>

Hello,

I’m hoping to re-raise the question because I haven’t made any progress towards figuring this particular phenomenon out.

The HuggingFace guide on tokenizers seems to imply that tokenizers are expected to be consistent, which is not what I am experiencing.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/a/3af0fcc25b3c58ebe67f01aee9f75dc395ff1737.png)  
[Summary of the tokenizers by HuggingFace](https://huggingface.co/docs/transformers/tokenizer_summary)

If anyone even knows whether the phenomenon pointed out in the initial post is normal or abnormal, please let me know!

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [April 22, 2024, 7:35am UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328/3 "2024-04-22T07:35:53Z")

</div>

Hey! This is because of the special token used by SentencePiece to mark the whitespace or the beginning of the word. You can [check here](https://github.com/google/sentencepiece?tab=readme-ov-file#what-is-sentencepiece) for more about how sentence piece algorithm works

In the case of “token”, those are actually two different tokens in str format and you can see that by converting ids to tokens. When you used the `.decode()` method it drops underlines.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('TeamUNIVA/Komodo_7B_v1.0.0')
>>> tokenizer.convert_ids_to_tokens(6029)
'▁token'
>>> tokenizer.convert_ids_to_tokens(5263)
'token'

```

---

<div class="post-metadata">

**Author:** ![Chahnwoo](https://avatars.discourse-cdn.com/v4/letter/c/a88e57/32.png) [@Chahnwoo](https://discuss.huggingface.co/u/Chahnwoo)\
**Post date:** [April 22, 2024, 8:12am UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328/4 "2024-04-22T08:12:54Z")

</div>

Thank you for that! I hadn’t taken that into consideration but that makes a lot of sense. I assume that is also part of why the same word may be tokenized differently at different points of a text.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [April 22, 2024, 8:13pm UTC](https://discuss.huggingface.co/t/tokenizer-mapping-the-same-token-to-multiple-token-ids/82328/5 "2024-04-22T20:13:18Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.