# Remove only certain special token id during tokenizer decode

**URL:** <https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436>\
**Category:** 🤗Tokenizers\
**Created:** [July 15, 2022, 7:42pm UTC](https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436 "2022-07-15T19:42:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![adibm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adibm/32/5390_2.png) [@adibm](https://discuss.huggingface.co/u/adibm)\
**Post date:** [July 15, 2022, 7:42pm UTC](https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436/1 "2022-07-15T19:42:39Z")

</div>

I am using a GPT2 based language model to generate some text. My training data has special tokens in them, so I want my model to generate those special tokens as well. The models generated text has a lot of padding token and I was wondering if there is a way to remove them during decoding. One way to solve it would be to pass it through a regular expression/filter and remove all the padding tokens.  
Here is an example of the generated text i got after i decoded it. I removed a lot of the pad tokens for this forum post.

```auto
'<|begincontext|><|user|>What can I go do I am bored sitting here.<|system|>What do you like to do and where would you like to do it?<|user|>I want something to do in or around NYC and Musical shows are one of my favorites.<|system|>I pulled up a list of 10 so far I think this one would interest you its on March 11th at 7:30 pm at the Kaufmann Concert Hall - Abbi Jacobson will be there.<|user|>That does interest me. Is there a direct bus I can take to get there?<|endcontext|>
\n\n<|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|><|pad|>FindAttractions<|endintent|>\n\n<|beginbelief|>Travel_goodForKids: True Travel_location: NYC<|endaction|>\n\n<|beginaction|>OFFER Travel_attractionName, OFFER Travel_category<|endintent|>\n\n<|beginresponse|>The Ac 9 Hotel By Marriott Brooklyn Bridge is a fine tourist spot.<|endintent|>\n\n<|pad|><|pad|>OFFER_attractionName, OFFER Travel_category<|endresponse|><|pad|><|pad|><|pad|><|pad|>'

```

---

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](https://discuss.huggingface.co/u/berkayberabi)\
**Post date:** [October 25, 2022, 8:54am UTC](https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436/2 "2022-10-25T08:54:00Z")

</div>

Hi, did you manage to find a proper solution for this except removing other special tokens manually in an iterative way?

---

<div class="post-metadata">

**Author:** ![adibm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adibm/32/5390_2.png) [@adibm](https://discuss.huggingface.co/u/adibm)\
**Post date:** [October 25, 2022, 5:13pm UTC](https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436/3 "2022-10-25T17:13:03Z")

</div>

unfortunately I was not able to find a proper solution to this.  
Initially I had used a regular expressions to remove certain strings but later I decided to tokenize the text and use a list comprehension to remove by token ids.

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 26, 2022, 5:24am UTC](https://discuss.huggingface.co/t/remove-only-certain-special-token-id-during-tokenizer-decode/20436/4 "2022-10-26T05:24:56Z")

</div>

Could you paste your code? Maybe it would be more details there.