# Added Tokens Not Decoding with Spaces

**URL:** <https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883>\
**Category:** 🤗Tokenizers\
**Created:** [October 19, 2021, 3:21am UTC](https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883 "2021-10-19T03:21:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcrchopra](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@mcrchopra](https://discuss.huggingface.co/u/mcrchopra)\
**Post date:** [October 19, 2021, 3:21am UTC](https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883/1 "2021-10-19T03:21:32Z")

</div>

Hi All,

**My goal is to add a set of starting tokens to a pre-trained AlbertTokenizerFast.**

In the Albert Pre-Trained Vocab (SentencePiece Model), all start tokens are preceded with the meta-symbol: ▁ (e.g. ▁hamburger).

I tried adding tokens, prefixed with the meta symbol:

```auto
new_tokens = [AddedToken("▁hamburger",), AddedToken("▁pizza")]
num_added_tokens = tokenizer.add_tokens(new_tokens)

```

However, [as this forum post shows](https://discuss.huggingface.co/t/add-new-tokens-for-subwords/489/3), input text to AddedToken is treated literally; so manually adding the meta-symbol prefixes doesn’t achieve the desired effect.

Instead, I tried using the single\_word parameter:

```auto
new_tokens = [AddedToken("hamburger", single_word=True, lstrip=True), AddedToken("pizza", single_word=True, lstrip=True)]
num_added_tokens = tokenizer.add_tokens(new_tokens)

```

This solution successfully encodes the new tokens where hamburger is being encoded by token 30001:

```auto
tokenizer('This hamburger tastes great') 
>> [2, 15, 30001, 53, 8, 345,3]

```

However, when I try to decode these ids, no space appears between “this” and “hamburger”:

```auto
tokenizer.decode([2, 15, 30001, 53, 8, 345,3]) 
>> ('Thishamburger tastes great')

```

I was wondering if anybody had any thoughts about how to fix this.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [October 19, 2021, 12:07pm UTC](https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883/2 "2021-10-19T12:07:20Z")

</div>

Does the same occur when setting `lstrip=False` when defining the new tokens?

---

<div class="post-metadata">

**Author:** ![mcrchopra](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@mcrchopra](https://discuss.huggingface.co/u/mcrchopra)\
**Post date:** [October 19, 2021, 4:19pm UTC](https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883/3 "2021-10-19T16:19:40Z")

</div>

Thank you for the response!

Yup, if I set lstrip=False, I see the same behavior:

```auto
tokenizer('This hamburger tastes great') 
>> [2, 15, 30001, 53, 8, 345,3]

```

```auto
tokenizer.decode([2, 15, 30001, 53, 8, 345,3]) 
>> ('Thishamburger tastes great')

```

Digging through the code, my hypothesis is:

- The [pre-tokenizer](https://github.com/huggingface/tokenizers/blob/master/tokenizers/src/pre_tokenizers/metaspace.rs#L55) replaces the meta-symbol with a space character, when applied to raw text.
- The decoder reverses the effects of the pre-tokenizer.
- When the decoder sees a token with a meta-symbol; it inserts a space (for correct viewing)
- Since the AddedTokens don’t go through the same pipeline (i.e. no metasymbol is added); I’m not sure if the pre-tokenizer is applied / if decoding works as expected.

Any thoughts on what could be going wrong? Or how one might approach this?

---

<div class="post-metadata">

**Author:** ![louisowen6](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@louisowen6](https://discuss.huggingface.co/u/louisowen6)\
**Post date:** [January 19, 2024, 5:49am UTC](https://discuss.huggingface.co/t/added-tokens-not-decoding-with-spaces/10883/4 "2024-01-19T05:49:09Z")

</div>

Hi @mcrchopra , I’m also facing the same issue. Are you able to solve this issue last time?