# Encoding and then decodeing text is not equal

**URL:** <https://discuss.huggingface.co/t/encoding-and-then-decodeing-text-is-not-equal/101851>\
**Category:** 🤗Tokenizers\
**Created:** [August 11, 2024, 9:08am UTC](https://discuss.huggingface.co/t/encoding-and-then-decodeing-text-is-not-equal/101851 "2024-08-11T09:08:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ron5569](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@ron5569](https://discuss.huggingface.co/u/ron5569)\
**Post date:** [August 11, 2024, 9:08am UTC](https://discuss.huggingface.co/t/encoding-and-then-decodeing-text-is-not-equal/101851/1 "2024-08-11T09:08:34Z")

</div>

I wonder why in some cases, encdoing the text and then decoding it, is not the same the original text  
For example very simple code

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

texts = [' A LoyaltyImport is a ...', ' You can ...', ' .withVariants("true")']
for row in texts:
    encoded = tokenizer(row,
                        add_special_tokens=False,
                        truncation=True,
                        padding=False,
                        max_length=2000,
                        return_overflowing_tokens=False,

                        return_length=False, )
    decoded = tokenizer.decode(encoded["input_ids"])
    if decoded != row:
        print("Different")
        print(row)
        print(decoded)

```

Then I see the output

```auto
Different
  A LoyaltyImport is a ...
  A LoyaltyImport is a...
Different
  You can ...
  You can...
Different
    .withVariants("true")
   .withVariants("true")

```

This is happen especially when using of dots and spaces.  
Why does it happen and how can I fix that  
Thanks!

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [August 12, 2024, 7:59am UTC](https://discuss.huggingface.co/t/encoding-and-then-decodeing-text-is-not-equal/101851/2 "2024-08-12T07:59:41Z")

</div>

Just add this line after loading the tokenizer:  
`tokenizer.clean_up_tokenization_spaces = False`

Llama3 has it to true by default, as you can see on its [tokenizer\_config.json](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json#L2054) file.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [August 13, 2024, 2:01pm UTC](https://discuss.huggingface.co/t/encoding-and-then-decodeing-text-is-not-equal/101851/3 "2024-08-13T14:01:13Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.