# When training Llama for sequence classification, should the final token be an EOS?

**URL:** <https://discuss.huggingface.co/t/when-training-llama-for-sequence-classification-should-the-final-token-be-an-eos/54500>\
**Category:** Beginners\
**Created:** [September 11, 2023, 4:57pm UTC](https://discuss.huggingface.co/t/when-training-llama-for-sequence-classification-should-the-final-token-be-an-eos/54500 "2023-09-11T16:57:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](https://discuss.huggingface.co/u/dblakely)\
**Post date:** [September 11, 2023, 4:57pm UTC](https://discuss.huggingface.co/t/when-training-llama-for-sequence-classification-should-the-final-token-be-an-eos/54500/1 "2023-09-11T16:57:59Z")

</div>

The [doc string](https://github.com/huggingface/transformers/blob/ce2e7ef3d96afaf592faf3337b7dd997c7ad4928/src/transformers/models/llama/modeling_llama.py#L906) says:

> LlamaForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do.

Should this “last token” be an EOS or simply the final token in the input without an EOS? My interpretation is that it should _not_ be an EOS, because otherwise, it would probably say that explicitly. Plus many people use the EOS as the pad token, in which case it would be identical to _not_ using the EOS as the “last token” for sequence classification.

However, I’m not certain so I’d appreciate it if anyone knew!

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](https://discuss.huggingface.co/u/dblakely)\
**Post date:** [September 18, 2023, 3:42pm UTC](https://discuss.huggingface.co/t/when-training-llama-for-sequence-classification-should-the-final-token-be-an-eos/54500/2 "2023-09-18T15:42:26Z")

</div>

@ArthurZ would you happen to know the answer to this? Thanks!

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](https://discuss.huggingface.co/u/ArthurZ)\
**Post date:** [September 18, 2023, 9:16pm UTC](https://discuss.huggingface.co/t/when-training-llama-for-sequence-classification-should-the-final-token-be-an-eos/54500/3 "2023-09-18T21:16:30Z")

</div>

This is a standard module, Llama was not trained for `LlamaForSequenceClassification` so you should use whatever you want no?