# Question regarding training of BartForConditionalGeneration

**URL:** <https://discuss.huggingface.co/t/question-regarding-training-of-bartforconditionalgeneration/4079>\
**Category:** Models\
**Created:** [March 2, 2021, 2:45pm UTC](https://discuss.huggingface.co/t/question-regarding-training-of-bartforconditionalgeneration/4079 "2021-03-02T14:45:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nbansal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbansal/32/49151_2.png) [@nbansal](https://discuss.huggingface.co/u/nbansal)\
**Post date:** [March 2, 2021, 2:45pm UTC](https://discuss.huggingface.co/t/question-regarding-training-of-bartforconditionalgeneration/4079/1 "2021-03-02T14:45:57Z")

</div>

Hello Guys,

I am trying to fine-tune the BART summarization model but due to the lack of big dataset, having some difficulties with the fine-tuning.

Thus, I decided to look at the trainig process of BartForConditionalGeneration model in detail. I came across this article titled ‘Introducing BART’ (sorry, only 2 links allowed for new users 😐) from one of the engineers, @sshleifer, at HuggingFace. It says that BartModel was directly fine-tuned for the summarisation task without **any new randomly initialized heads**.

**My question is about this fine-tuning process, especially on CNN-DailyMail dataset. Do you guys fine-tune the entire Bart model or only the decoder or something else?**

I looked at the example [fine-tuning script](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py) provided on the GitHub but I didn’t find anything related to freezing some part of the model.

&nbsp;

I also tried to look at the source code of the BartForConditionalGeneration model and observed the following -

Its just adds a [linear layer](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1207) on top of the BartModel (copy-pasting the ` __init__ ` code here for quick reference).

```auto
self.model = BartModel(config)
self.register_buffer("final_logits_bias", torch.zeros((1, self.model.shared.num_embeddings)))
self.lm_head = nn.Linear(config.d_model, self.model.shared.num_embeddings, bias=False)

```

At first, I thought these are the new parameters that are being introduced and thus, being trained. Therefore, I tried the following code to check the number of trainable parameters while keeping the endoer and decoder fixed -

```auto
from transformers import BartModel, BartForConditionalGeneration, BartTokenizer

def freeze_params(model):
    for par in model.parameters():
        par.requires_grad = False

model_sum = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
freeze_params(model_sum.get_encoder()) ## freeze the encoder
freeze_params(model_sum.get_decoder()) ## freeze the decoder 

model_sum.train() ## set the train mode
train_p = [p for p in model_sum.parameters() if p.requires_grad] ## get the trainable params
print(f'Length of train params in Summarization Model : {len(train_p)}')

```

But this code shows that the list is empty. One thing I can do is to explictly set the `requires_grad=True` for the paramters in the `model_sum.lm_head` and only fine-tune these parameters. But I am curious to understand the original training/fine-tuning process. **It would be of great help to me if you guys could answer my question.**

Thanks,  
Naman

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [March 2, 2021, 3:05pm UTC](https://discuss.huggingface.co/t/question-regarding-training-of-bartforconditionalgeneration/4079/2 "2021-03-02T15:05:41Z")

</div>

I answered on github: [Question regarding training of BartForConditionalGeneration · Issue #10479 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/10479)
