# How to decode GPT2

**URL:** <https://discuss.huggingface.co/t/how-to-decode-gpt2/16160>\
**Category:** 🤗Transformers\
**Created:** [March 28, 2022, 12:20pm UTC](https://discuss.huggingface.co/t/how-to-decode-gpt2/16160 "2022-03-28T12:20:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mwitiderrick](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mwitiderrick/32/17324_2.png) [@mwitiderrick](https://discuss.huggingface.co/u/mwitiderrick)\
**Post date:** [March 28, 2022, 12:20pm UTC](https://discuss.huggingface.co/t/how-to-decode-gpt2/16160/1 "2022-03-28T12:20:40Z")

</div>

What’s the proper way to decode the output of GPT2

```auto
from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
tokenizer.decode(output)

```

This line `tokenizer.decode(output)` gives me this error

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7569067dabf0cdc35cd55afa1b6c67715cb3af29.png)

---

<div class="post-metadata">

**Author:** ![jdwx](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdwx/32/4307_2.png) [@jdwx](https://discuss.huggingface.co/u/jdwx)\
**Post date:** [March 28, 2022, 6:55pm UTC](https://discuss.huggingface.co/t/how-to-decode-gpt2/16160/2 "2022-03-28T18:55:04Z")

</div>

I don’t think the code that you’ve written will give you anything that a tokenizer can decode. By calling TFGPT2Model without a task head (e.g., TFGPTLMHeadModel for causal language modeling), you’re just getting back a TFBaseModelOutputWithPastAndCrossAttentions object with GPT-2’s 768-dimension word embeddings for each input token in output.last\_hidden\_state.

In your case, output.last\_hidden\_state is a tensor with shape (1, 10, 768) because you have one input with 10 tokens, and GPT-2 uses 768 embedding dimensions.

The HuggingFace model is to add a “modelling head” on top of the base model to help perform whatever NLP task you’re after. If you’re looking to get tokens you can decode, that’s probably causal language modelling.

A simple TensorFlow example for causal language modelling might look like:

```auto
from transformers import GPT2Tokenizer, TFGPT2LMHeadModel

def main():
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    text = "Replace me by any text you'd like."
    encoded_input = tokenizer(text, return_tensors='tf')
    model = TFGPT2LMHeadModel.from_pretrained('gpt2')
    output = model.generate(**encoded_input)
    decoded = tokenizer.decode(output[0])
    print(decoded)

if __name__ == ' __main__':
    main()

```

In this example, model.generate() is doing a _lot_ of heavy lifting for you over calling model(encoded\_input), most of which is controllable by its enormous number of additional parameters.

---

<div class="post-metadata">

**Author:** ![jdwx](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdwx/32/4307_2.png) [@jdwx](https://discuss.huggingface.co/u/jdwx)\
**Post date:** [March 28, 2022, 7:48pm UTC](https://discuss.huggingface.co/t/how-to-decode-gpt2/16160/3 "2022-03-28T19:48:58Z")

</div>

For the sake of completeness, here’s a minimal example that does call the model directly (i.e. without generate()):

```auto
import tensorflow as tf
from transformers import GPT2Tokenizer, TFGPT2LMHeadModel

def main():
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    text = "When it comes to making transformers easy, HuggingFace is the"
    encoded_input = tokenizer(text, return_tensors='tf')
    model = TFGPT2LMHeadModel.from_pretrained('gpt2')
    output = model(encoded_input)
    logits = output.logits[0, -1, :]
    softmax = tf.math.softmax(logits, axis=-1)
    argmax = tf.math.argmax(softmax, axis=-1)
    print(text, "[", tokenizer.decode(argmax), "]")

if __name__ == ' __main__':
    main()

```

It generates exactly one word, which in this case should be “best” since it’s deterministically picking the highest probabibilty word in the output:

```auto
2022-03-28 15:45:30.783765: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
All model checkpoint layers were used when initializing TFGPT2LMHeadModel.

All the layers of TFGPT2LMHeadModel were initialized from the model checkpoint at gpt2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.
When it comes to making transformers easy, HuggingFace is the [best]

```

---

<div class="post-metadata">

**Author:** ![Frankb](https://avatars.discourse-cdn.com/v4/letter/f/278dde/32.png) [@Frankb](https://discuss.huggingface.co/u/Frankb)\
**Post date:** [June 17, 2022, 1:58am UTC](https://discuss.huggingface.co/t/how-to-decode-gpt2/16160/4 "2022-06-17T01:58:07Z")

</div>

it seems that the softmax is redundancy when only the best token is required, because it doesn’t influence the order of logits.