# Can I use "AutoModel For Sequence Classification" class for generative models?

**URL:** <https://discuss.huggingface.co/t/can-i-use-automodel-for-sequence-classification-class-for-generative-models/43244>\
**Category:** 🤗Transformers\
**Created:** [June 14, 2023, 2:54pm UTC](https://discuss.huggingface.co/t/can-i-use-automodel-for-sequence-classification-class-for-generative-models/43244 "2023-06-14T14:54:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomad01](https://avatars.discourse-cdn.com/v4/letter/t/dec6dc/32.png) [@tomad01](https://discuss.huggingface.co/u/tomad01)\
**Post date:** [June 14, 2023, 2:54pm UTC](https://discuss.huggingface.co/t/can-i-use-automodel-for-sequence-classification-class-for-generative-models/43244/1 "2023-06-14T14:54:15Z")

</div>

I want to use AutoModelForSequenceClassification with a llama 7b model  
How will the input flow in the model if I load the model with this class ?

---

<div class="post-metadata">

**Author:** ![WANGYIWEI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wangyiwei/32/19668_2.png) [@WANGYIWEI](https://discuss.huggingface.co/u/WANGYIWEI)\
**Post date:** [September 20, 2023, 3:06pm UTC](https://discuss.huggingface.co/t/can-i-use-automodel-for-sequence-classification-class-for-generative-models/43244/2 "2023-09-20T15:06:00Z")

</div>

Yes it works, I used a GPT2-XL to do NLU tasks.  
But it also depends on the weights you use whether it has a class called `transformers.XXXForSequenceClassification`, where XXX is the model class/ name.  
Such as here: [https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/models/gpt2/modeling\_gpt2.py#L1376](https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/models/gpt2/modeling_gpt2.py#L1376)

---

<div class="post-metadata">

**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\
**Post date:** [April 15, 2024, 9:24am UTC](https://discuss.huggingface.co/t/can-i-use-automodel-for-sequence-classification-class-for-generative-models/43244/3 "2024-04-15T09:24:35Z")

</div>

Hi, did you try this? Does it give better results as compared to the BERT models? I am trying the same, but the model is overfitting on the training dataset.