# Longformer for Encoder Decoder with gradient checkpointing

**URL:** <https://discuss.huggingface.co/t/longformer-for-encoder-decoder-with-gradient-checkpointing/13428>\
**Category:** Beginners\
**Created:** [January 6, 2022, 10:40am UTC](https://discuss.huggingface.co/t/longformer-for-encoder-decoder-with-gradient-checkpointing/13428 "2022-01-06T10:40:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![LadyHangaku](https://avatars.discourse-cdn.com/v4/letter/l/71c47a/32.png) [@LadyHangaku](https://discuss.huggingface.co/u/LadyHangaku)\
**Post date:** [January 6, 2022, 10:40am UTC](https://discuss.huggingface.co/t/longformer-for-encoder-decoder-with-gradient-checkpointing/13428/1 "2022-01-06T10:40:53Z")

</div>

I’m struggling to find the right transformers class for my task.  
I want to solve a seq2seq problem with an encoder decoder longformer. I generated one with this [german RoBERTa model](https://huggingface.co/benjamin/roberta-base-wechsel-german) using [this script](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb).  
I know that I could use [`EncoderDecoderModel()`](https://huggingface.co/docs/transformers/model_doc/encoderdecoder#transformers.EncoderDecoderModel), but the issue is that it doesn’t support gradient checkpointing, which I desperately need, because otherwise it wouldn’t run on the machine.  
And if I understand it correctly, the class [`LEDModel()`](https://huggingface.co/docs/transformers/v4.15.0/en/model_doc/led#transformers.LEDModel) only takes already built encoder decoder models and not just a plain longformer to chain it together, so that is also not an option.  
I thought about initializing two seperate Longformers for encoder and decoder with `LongformerModel()`, but then I don’t know how to glue them together. Can someone explain how it works?  
Or does anyone have another suggestions on how I can solve this problem?  
Thank you very much!

---

<div class="post-metadata">

**Author:** ![LadyHangaku](https://avatars.discourse-cdn.com/v4/letter/l/71c47a/32.png) [@LadyHangaku](https://discuss.huggingface.co/u/LadyHangaku)\
**Post date:** [January 7, 2022, 10:41am UTC](https://discuss.huggingface.co/t/longformer-for-encoder-decoder-with-gradient-checkpointing/13428/2 "2022-01-07T10:41:04Z")

</div>

I found a solution which at least helps a little:  
When using `EncoderDecoderModel()`, it is possible to set gradient checkpointing at least on the encoder part:  
`model.encoder.config.gradient_checkpointing = True`