# RobertaTokenizer decode and tokenize do not have the same output

**URL:** <https://discuss.huggingface.co/t/robertatokenizer-decode-and-tokenize-do-not-have-the-same-output/59709>\
**Category:** 🤗Tokenizers\
**Created:** [October 24, 2023, 2:43pm UTC](https://discuss.huggingface.co/t/robertatokenizer-decode-and-tokenize-do-not-have-the-same-output/59709 "2023-10-24T14:43:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BettyFbr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bettyfbr/32/20595_2.png) [@BettyFbr](https://discuss.huggingface.co/u/BettyFbr)\
**Post date:** [October 24, 2023, 2:43pm UTC](https://discuss.huggingface.co/t/robertatokenizer-decode-and-tokenize-do-not-have-the-same-output/59709/1 "2023-10-24T14:43:44Z")

</div>

I use a `RobertaTokenizer` to tokenize sentences that contains french characters like é or ç.  
I need the generated tokens with the `Ġ` character and the french characters well formatted.

For instance with the input  
`input = "3 allées paris, 75000"`  
`[tokenizer.decode([token]) for token in tokenizer.encode(input)]` outputs `['<s>', ' 3', ' all', 'ées', ' paris', ',', ' 7', '5000', '</s>']` so the `Ġ` are replaced by spaces.  
And `tokenizer.tokenize(input)` outputs `['Ġ3', 'Ġall', 'ées', 'Ġparis', ',', 'Ġ7', '5000']` so the french characters are not well formatted.

I used to do this, and it used to work:

```auto
inputs = self.tokenizer.encode_plus(input, return_tensors="pt")
ids = inputs['input_ids'].cpu().tolist()
clean_tokens = [self.tokenizer.decode([token]) for token in ids[0]]

```

But for some reasons I cannot understand, it does not output the tokens with the Ġ characters anymore and I cannot figure out what was the breaking change.  
Do you have any idea ?