# What does from\_pretrained do?

**URL:** <https://discuss.huggingface.co/t/what-does-from-pretrained-do/28126>\
**Category:** Beginners\
**Created:** [December 19, 2022, 3:31pm UTC](https://discuss.huggingface.co/t/what-does-from-pretrained-do/28126 "2022-12-19T15:31:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkabra](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@mkabra](https://discuss.huggingface.co/u/mkabra)\
**Post date:** [December 19, 2022, 3:31pm UTC](https://discuss.huggingface.co/t/what-does-from-pretrained-do/28126/1 "2022-12-19T15:31:13Z")

</div>

Hi, I am trying to visualize what **from\_pretrained** do  
For eg:

```auto
from transformers import BertModel
finbert_bertmodel=BertModel.from_pretrained('ProsusAI/finbert')

```

#The Bert Model transformer outputting raw hidden-states without any specific head on top.  
#While Finbert has BertForSequenceClassification architecture

This is my understanding of from\_pretrained for this piece of code. Here, it loads all the corresponding weights of Finbert into the architecture of BertModel.  
Similarly, for a different model, say:

`finbert_maskedLM=AutoModelForMaskedLM.from_pretrained('ProsusAI/finbert')`

Weights of Finbert will be loaded into the architecture of MaskedLM, the weights of last layer of MaskedLM will be randomly initialised as Finbert architecture is of Sequence classification, and not of MaskedLM.  
Is my understanding correct?  
Please add additional details and any link which has additional information.

---

<div class="post-metadata">

**Author:** ![mkabra](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@mkabra](https://discuss.huggingface.co/u/mkabra)\
**Post date:** [December 20, 2022, 5:01pm UTC](https://discuss.huggingface.co/t/what-does-from-pretrained-do/28126/2 "2022-12-20T17:01:22Z")

</div>

@nielsr @sgugger ,

Can you please check this question ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 10, 2024, 8:37am UTC](https://discuss.huggingface.co/t/what-does-from-pretrained-do/28126/3 "2024-09-10T08:37:02Z")

</div>

Hi,

So [ProsusAI/finbert · Hugging Face](https://huggingface.co/ProsusAI/finbert) indeed has a `BertForSequenceClassification` architecture as can be seen from the “architectures” field in config.json.

When you do:

```python
from transformers import BertModel
finbert_bertmodel=BertModel.from_pretrained('ProsusAI/finbert')

```

then what happens is, all weights of the BERT model get initialized from the checkpoint on the hub, and the weights of the sequence classifier on top (which the finbert repo included) will be discarded, as BertModel does not include a sequence classification head on top.

When you do:

```python
from transformers import BertForMaskedLM
finbert_bertmodel=BertForMaskedLM.from_pretrained('ProsusAI/finbert')

```

then all weights of the BERT model get initialized from the checkpoint on the hub, the sequence classifier weights gets discarded, and a randomly initialized language modeling head on top will be added.