# How to extract encoding before classification layer?

**URL:** <https://discuss.huggingface.co/t/how-to-extract-encoding-before-classification-layer/32216>\
**Category:** 🤗Transformers\
**Created:** [February 21, 2023, 8:38pm UTC](https://discuss.huggingface.co/t/how-to-extract-encoding-before-classification-layer/32216 "2023-02-21T20:38:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![SpaceHunterInf](https://avatars.discourse-cdn.com/v4/letter/s/43a26b/32.png) [@SpaceHunterInf](https://discuss.huggingface.co/u/SpaceHunterInf)\
**Post date:** [February 21, 2023, 8:38pm UTC](https://discuss.huggingface.co/t/how-to-extract-encoding-before-classification-layer/32216/1 "2023-02-21T20:38:36Z")

</div>

Greetings,

So I am using a RobertaForSequenceClassification model for natural language inference ([ynie/roberta-large-snli\_mnli\_fever\_anli\_R1\_R2\_R3-nli · Hugging Face](https://huggingface.co/ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli/)). How may I obtain the encoding of my sequence before it passes through the final classification layer?

Here are what I have tried:  
Firstly I identified all the parameters  
[ …  
‘roberta.encoder.layer.23.output.LayerNorm.weight’,  
‘roberta.encoder.layer.23.output.LayerNorm.bias’,  
‘classifier.dense.weight’,  
‘classifier.dense.bias’,  
‘classifier.out\_proj.weight’,  
‘classifier.out\_proj.bias’]

Then I use the hidden state flag to obtain the hidden state vectors with a single input (batch\_size =1).  
`outputs =model(input_ids,attention_mask=attention_mask,token_type_ids=token_type_ids,output_hidden_states=True, labels=None)`  
`outputs[1]` is the hidden states which has size `[1,23,1024]`, 1024 is my hidden state dim

```auto
t = []
matrix = None
bias = None
dense_m = None
dense_b = None

for name, param in model.named_parameters():
    if name == 'classifier.dense.weight':
        dense_m = param
    if name == 'classifier.dense.weight':
        dense_b = param
    if name == 'classifier.out_proj.weight':
        matrix = param
    if name == 'classifier.out_proj.bias':
        bias = param

test = outputs[1][-1][-1][-1]
t = torch.mm(dense_m, torch.unsqueeze(test, dim=1)).T + dense_b
print(torch.mm(matrix, t).T + bias)
print(outputs[0])

```

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7da9594e8dd5a6be69fa80516ede4f5e53f6abaa.png)

The final output has shape `[1,3]`, but mine is `[23,3]` and the last row is not equal to the original output. What have I done wrong or what should I do in this case?

Thank you very much!