# Mask only specific words

**URL:** <https://discuss.huggingface.co/t/mask-only-specific-words/173>\
**Category:** 🤗Tokenizers\
**Created:** [July 10, 2020, 8:47am UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173 "2020-07-10T08:47:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gabrer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gabrer/32/113_2.png) [@Gabrer](https://discuss.huggingface.co/u/Gabrer)\
**Post date:** [July 10, 2020, 8:47am UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173/1 "2020-07-10T08:47:59Z")

</div>

What would be the best strategy to **mask only specific words** during the LM training?  
My aim is to mask only words of interest which I have previously collected in a list.

The issue arises since the tokenizer, not only splits a single word in multiple tokens, but it also adds special characters if the word does not occur at the begging of a sentence.  
E.g.:

> The word “ **Valkyria** ”:
> 
> - at the _beginning_ of a sentences gets split as **[‘V’, ‘alky’, ‘ria’]** with corresponding IDs: **[846, 44068, 6374]**.
> - while in the _middle_ of a sentence as **[‘ĠV’, ‘alky’, ‘ria’]** with corresponding IDs: **[468, 44068, 6374]**,

This is just one of the issues forcing me to have multiple entries in my list of to-be-filtered IDs.

I have already had a look at the `mask_tokens()` function into the `DataCollatorForLanguageModeling class`, which is the function actually masking the tokens during each batch, but I cannot find any efficient and smart way to mask only specific words and their corresponding IDs.

---

<div class="post-metadata">

**Author:** ![anthony](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anthony/32/2505_2.png) [@anthony](https://discuss.huggingface.co/u/anthony)\
**Post date:** [July 10, 2020, 3:38pm UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173/2 "2020-07-10T15:38:51Z")

</div>

When using the “fast” variant of the tokenizers available in [huggingface/transformers](https://github.com/huggingface/transformers), whenever you encode some text, you get back a [BatchEncoding](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=batchencoding#transformers.BatchEncoding).  
This `BatchEncoding` provides some helpful mappings that we can use in this kind of situation. So, you should be able to:

1. Find the word associated with any token using [token\_to\_word](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=batchencoding#transformers.BatchEncoding.token_to_word). This method returns the index of the word in the input sequence.
2. Once you know the word’s index, you can actually retrieve its span with [word\_to\_chars](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=batchencoding#transformers.BatchEncoding.word_to_chars). This will let you extract the word from the input sequence.

---

<div class="post-metadata">

**Author:** ![Gabrer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gabrer/32/113_2.png) [@Gabrer](https://discuss.huggingface.co/u/Gabrer)\
**Post date:** [July 11, 2020, 10:31am UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173/3 "2020-07-11T10:31:30Z")

</div>

Hi @Anthony,

Thank you for your prompt reply!  
The approach you proposed, if I’m not mistaken, would be helpful in permanently masking words when reading the dataset.

Instead, I am interested in dynamically masking words at batch time.  
To be more clear, I would like to implement (efficiently) a [mask\_tokens()](https://github.com/huggingface/transformers/blob/3dcb748e31be8c7c9e4f62926c5c144c62d07218/src/transformers/data/data_collator.py#L169) function (as the one defined in the `DataCollatorForLanguageModeling` class) which masks only IDs corresponding to words provided in a specific list.  
These IDs would not be masked during each batch but following some stochastic strategy, unlike the approach which would mask them when the dataset is read at the beginning.

I am wondering whether there is an efficient and neat way to do so, exploiting the Huggingface functions and taking also into account that some words get split in multiple IDs depending on the tokenizer at hand.

---

<div class="post-metadata">

**Author:** ![RylanSchaeffer](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RylanSchaeffer](https://discuss.huggingface.co/u/RylanSchaeffer)\
**Post date:** [November 6, 2021, 7:30pm UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173/4 "2021-11-06T19:30:26Z")

</div>

@Gabrer did you find a solution?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [November 7, 2021, 12:30pm UTC](https://discuss.huggingface.co/t/mask-only-specific-words/173/5 "2021-11-07T12:30:46Z")

</div>

You can create your own custom mask and merge that with the automatic mask.

```python
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
word = "Valkyria"
sentences = ["Valkyria at the beginning of sentence", "And here Valkyria in the middle of one."]
# Index of word in the sentences (word-tokenized!)
word_idxs_in_sent = [sent.split(" ").index(word) for sent in sentences]

encoded = tokenizer(sentences, return_tensors="pt", padding=True)
print("Original mask", encoded["attention_mask"])

# For each sentence, set a subword token to False if it belongs to the word (becomes 0 in LongTensor)
match_idxs = torch.LongTensor([[wid != word_idxs_in_sent[batch_idx] for wid in encoded.word_ids(batch_idx)]
              for batch_idx in range(len(sentences))])
print("Subword indices of matching word", match_idxs)

# Merge: if a word is zero in our custom match, merge, if not, use the original mask
# This ensures that we mask the word IDs but keep the original mask for special tokens (cls, pad, etc.)
merged = torch.where(match_idxs == 0, match_idxs, encoded["attention_mask"])
print("Merged mask", merged)

```

Results:

```auto
original mask tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
Subword indices of matching word tensor([[1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])
Merged mask tensor([[1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])

```