# Why multilingual BERT tokenizer doesn't remove accent markers?

**URL:** <https://discuss.huggingface.co/t/why-multilingual-bert-tokenizer-doesnt-remove-accent-markers/8468>\
**Category:** 🤗Tokenizers\
**Created:** [July 18, 2021, 7:25am UTC](https://discuss.huggingface.co/t/why-multilingual-bert-tokenizer-doesnt-remove-accent-markers/8468 "2021-07-18T07:25:32Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xuxuxu](https://avatars.discourse-cdn.com/v4/letter/x/b19c9b/32.png) [@xuxuxu](https://discuss.huggingface.co/u/xuxuxu)\
**Post date:** [July 18, 2021, 7:25am UTC](https://discuss.huggingface.co/t/why-multilingual-bert-tokenizer-doesnt-remove-accent-markers/8468/1 "2021-07-18T07:25:32Z")

</div>

Hi everyone  
I’m wondering why the pretrained ‘bert-base-multilingual-cased’ tokenizer cannot remove accent markers?

To reproduce:

```auto
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
tokenizer.convert_ids_to_tokens(tokenizer.encode('résumé'))

```

and the result gives

```auto
['[CLS]', 'r', '##és', '##um', '##é', '[SEP]’]

```

However, the document says “When calling encode() or encode\_batch(), the input text(s) go through the following pipiline: Normalization, Pre-Tokenization, The Model, Post-Processing” in this page: [The tokenization pipeline — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/pipeline.html). The Normalization includes stripping whitespace, removing accented characters or lowercasing all text, which is also mentioned in the BERT GitHub README: [bert/multilingual.md at master · google-research/bert · GitHub](https://github.com/google-research/bert/blob/master/multilingual.md).  
According to the huggingface tokenizer introduction, we can evaluate as follows

```auto
from tokenizers import normalizers 
from tokenizers.normalizers import NFD, StripAccents 
normalizer = normalizers.Sequence([NFD(), StripAccents()])
normalizer.normalize_str('résumé')

```

and the result gives:

```auto
'resume’

```

Thus, it seems to me that `tokenizer` does not perform accent removal.  
What’s happening here? I’m quite confused.

Please help clarify this problem.