# What is the magic behind BartForConditionalGeneration？

**URL:** <https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184>\
**Category:** Models\
**Created:** [March 29, 2021, 7:27am UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184 "2021-03-29T07:27:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\
**Post date:** [March 29, 2021, 7:27am UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/1 "2021-03-29T07:27:15Z")

</div>

For some reason, I want to modify the linear layer inside BartForConditionalGeneration. Therefore, I use a BartModel with Linear just like BartForConditionalGeneration. The Performance has a large drop-down when using BartModel with Linear. It’s so strange 😭 😢

For same training and evaluation data:  
BartForConditionalGeneration  
`{'Bleu_1': 0.3756316307557612, 'Bleu_2': 0.2187763449001214, 'Bleu_3': 0.14257622050968358, 'Bleu_4': 0.09772224033332834, 'ROUGE_L': 0.31379157899331667, 'CIDEr': 0.2487453519966872}`  
BartModel with Linear  
`{'Bleu_1': 0.28135212418299216, 'Bleu_2': 0.039374791862140796, 'Bleu_3': 5.7869382968790495e-08, 'Bleu_4': 9.583990840791874e-11, 'ROUGE_L': 0.13023605134624447, 'CIDEr': 0.012828799693149772}`

Here is my code  
[BartForConditionalGeneration](https://colab.research.google.com/drive/1Xq9wTspK-I7fVviny_3NNMX1RBAC6oXS?usp=sharing)  
[BartModel with Linear](https://drive.google.com/file/d/1hOlU5jtxyBYiV0QNeihzQEJsPL071QpD/view?usp=sharing)

Some trial and notes for your reference:

- use set\_output\_embeddings to replace linear layer - dropdown
- tie linear weight with BartModel.shared weight - dropdown
- re-init linear weight with config.std and 0 mean - dropdown
- clone BartModel.shared weight to linear weight - dropdown
- add bias or remove bias - dropdown
- extend BartForConditionalGeneration and rename lm\_head model - dropdown
- use different seq2seq models (t5) - dropdown

---

<div class="post-metadata">

**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\
**Post date:** [March 29, 2021, 3:31pm UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/2 "2021-03-29T15:31:52Z")

</div>

For copy shard weight or tie shared.weight, the result will be little bit better, but still far from BartForConditionalGeneration:

## Copy weight:

> **[Google Colab](https://colab.research.google.com/drive/1Oslg-N3u6lWYjE9pyKB7WtEwYS5Aabn7?usp=sharing)**

```auto
import copy
lm_head = nn.Linear(pretrained.config.hidden_size, tokenizer. __len__ (), bias=False).to(device)
lm_head.weight = copy.copy(pretrained.shared.weight)

```

Result:  
`{'Bleu_1': 0.3009317871068947, 'Bleu_2': 0.15865498886231086, 'Bleu_3': 0.09005394179103642, 'Bleu_4': 0.05191279861663496, 'ROUGE_L': 0.22372818945128858, 'CIDEr': 0.15579250859745616}`

## Tie weight:

> **[Google Colab](https://colab.research.google.com/drive/1bS9phjEZiSizgBXofmHUEmWO9VpGd-GM?usp=sharing)**

```auto
lm_head = nn.Linear(pretrained.config.hidden_size, tokenizer. __len__ (), bias=False).to(device)
lm_head.weight = pretrained.shared.weight

```

Result：  
·{‘Bleu\_1’: 0.30315688210424285, ‘Bleu\_2’: 0.1590543852533103, ‘Bleu\_3’: 0.08880157836836094, ‘Bleu\_4’: 0.04979010468389569, ‘ROUGE\_L’: 0.22960729767442484, ‘CIDEr’: 0.1570861241454517}·

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [March 29, 2021, 4:13pm UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/3 "2021-03-29T16:13:35Z")

</div>

In BartForConditionalGeneration a fixed bias is included as well.

> <https://github.com/huggingface/transformers/blob/06a6fea7820dc3e89d09430a49bce1c72b173647/src/transformers/models/bart/modeling_bart.py#L1209>

> <https://github.com/huggingface/transformers/blob/06a6fea7820dc3e89d09430a49bce1c72b173647/src/transformers/models/bart/modeling_bart.py#L1293>

---

<div class="post-metadata">

**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\
**Post date:** [March 29, 2021, 4:52pm UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/4 "2021-03-29T16:52:05Z")

</div>

> [@BramVanroy](#):
>
> `self.final_logits_bias`

I have tried print final\_logits\_bias during training and evaluation, it remain zero all the time. So it should be fine to ignore it ?

---

<div class="post-metadata">

**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\
**Post date:** [March 29, 2021, 5:18pm UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/5 "2021-03-29T17:18:01Z")

</div>

When manually decode the result from custom linear, it‘s different from the model.generate result.

From [BartModel with Linear](https://drive.google.com/file/d/1hOlU5jtxyBYiV0QNeihzQEJsPL071QpD/view?usp=sharing)

```auto
demo = """Looking good , feeling good Born to a model mom and a suit maker dad , fashion was actually in my blood . I always had a strong desire to dress in a certain way and to stand out from the crowd . I made my own toys when I was a young child and sewed my first skirt at just 10 years old . A friend 's mother took one look at my skirt and told me that I should be a patternmaker . In high school I started making my own clothes , mostly changing other things because I never liked anything how it was when I bought it . During the last two years of school , I worked part - time for a small business that made hand - painted silk clothing and bags . The owner became the teacher who got me into design in the first place . Another useful bit of work experience then came when I worked at a showroom during fashion week and found it very exciting . From there I worked at a top clothing store while I got my business started . For my business I started out with the idea that everything I did would be hand - made and one - of - a - kind , specially made for one individual who hopefully had the same tastes as me . Every morning I jumped out of bed , went to my studio and worked on my projects . This just showed how enthusiastic I felt about my work . And at night I even dreamed of new designs ! Fashion design is _ art . What I mean is that it 's something close to you and something you can touch and feel , and actually interact with . My advice to any young person who wants to be a fashion designer is to get the basic skills early on , such as sewing and pattern - making . Even if you end up specializing , it 's really important to understand all aspects of design in order to make high - quality clothes . Also , if you dream of having your own clothing line , the best thing to do is start wearing your clothes . You have to try and do this because that 's the way you 're going to develop something that 's all yours and unlike anyone else 's . I passionately believe that the right clothing can make people feel better and give them more confidence . </s> When the author was in high school , she </s> began to make clothes on her own"""
pretrained.eval()
lm_head.eval()
input_ids = tokenizer.encode(demo,return_tensors='pt',add_special_tokens=False).to(device)
tokenizer.decode(torch.argmax(lm_head(pretrained(input_ids).last_hidden_state),-1)[0],skip_special_tokens=True)

```

Result:  
`made her to to make`

However,

```auto
from transformers import AutoTokenizer, BartForConditionalGeneration
from transformers import pipeline
hf_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base').to('cuda')

hf_model.lm_head = lm_head
hf_model.model = pretrained

result = hf_model.generate(input_ids,do_sample=False,top_k=1,num_beams=1)
print(result)
tokenizer.decode(result[0],skip_special_tokens=True)

```

Result  
`made her to make clothes`

---

<div class="post-metadata">

**Author:** ![p208p2002](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/p208p2002/32/17321_2.png) [@p208p2002](https://discuss.huggingface.co/u/p208p2002)\
**Post date:** [March 30, 2021, 3:42am UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/6 "2021-03-30T03:42:45Z")

</div>

please note that the behavior(shift\_tokens\_right) in BartModel and BartForConditionalGeneration is different

> <https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_bart.py#L1140>

> <https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_bart.py#L1274>

you should consider manual provide `decoder_input_ids`, and you can get the correct results

 ![擷取](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/13672f6213a73b969b29ad201542349cfe2f4a11.png)

---

<div class="post-metadata">

**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\
**Post date:** [March 30, 2021, 6:22am UTC](https://discuss.huggingface.co/t/what-is-the-magic-behind-bartforconditionalgeneration/5184/7 "2021-03-30T06:22:13Z")

</div>

I believe this is the reason. Thank you for pointing it out. 🥰