# Error fine tuning Donut model using LoRA **URL:** \ **Category:** Beginners\ **Created:** [June 1, 2025, 10:32pm UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587 "2025-06-01T22:32:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![DenisMir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denismir/32/48618_2.png) [@DenisMir](https://discuss.huggingface.co/u/DenisMir)\ **Post date:** [June 1, 2025, 10:33pm UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/1 "2025-06-01T22:33:00Z")
Hello, I’m new to ML and this is probably a basic problem. I’m trying to fine tune Donut base model using my documents but getting errors. > **[Anaconda | Built to Advance Open Source AI](https://anaconda.com/app/share/notebooks/98670ba2-545f-4554-bc6a-30e277b1d710/overview)** > > Anaconda simplifies, safeguards, and accelerates open-source AI with a trusted platform, enabling secure scaling, real-time insights, and community collaboration. The error is TypeError: DonutSwinModel.forward() got an unexpected keyword argument ‘input\_ids’ I’m generating a dataset using document images and annotations.jsonl with following data {“label”: “{"load\_id": "1234", "carrier\_name": "Bison"}”, “image”: “TOUR\_LOGISTICS\_0.png”} My dataset has { “pixel\_values”: batch[“pixel\_values”], “decoder\_input\_ids”: batch[“decoder\_input\_ids”], “labels”: batch[“labels”] } Isn’t Trainer process knows which field to use for Encoder and Decoder? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 2, 2025, 4:03am UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/2 "2025-06-02T04:03:07Z")
It may be a version-dependent bug in Transoformers… > **[naver-clova-ix/donut-base · Error Fine Tuning due to unexpected keyword argument](https://huggingface.co/naver-clova-ix/donut-base/discussions/13)** > > I was constantly getting the following error while trying to train the donut model. > > > \### System Info > > Reopening: https://github.com/huggingface/transformers/issues/3…5838 > > \### Who can help? > > \_No response\_ > > \### Information > > \- \[\] The official example scripts > \- \[\] My own modified scripts > > \### Tasks > > \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...) > \- \[\] My own task or dataset (give details below) > > \### Reproduction > > see https://github.com/huggingface/transformers/issues/35838 > > \### Expected behavior > > see https://github.com/huggingface/transformers/issues/35838 ---
**Author:** ![DenisMir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denismir/32/48618_2.png) [@DenisMir](https://discuss.huggingface.co/u/DenisMir)\ **Post date:** [June 2, 2025, 12:47pm UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/3 "2025-06-02T12:47:28Z")
I downgraded to transformers==4.45.2 and it didn’t help. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 3, 2025, 12:08am UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/4 "2025-06-03T00:08:38Z")
Hmm… Maybe the wrong `collate_fn` is assigned… Or maybe the model settings are wrong. > **[Debugging the training pipeline - Hugging Face LLM Course](https://huggingface.co/learn/llm-course/en/chapter8/4)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. [https://stackoverflow.com/questions/77182311/question-about-data-collator-throwing-a-key-error-in-hugging-face](https://stackoverflow.com/questions/77182311/question-about-data-collator-throwing-a-key-error-in-hugging-face) > > > Hello, > > Thank you again for the fantastic work on this library and all the exa…mples you are including !! > Big up @younesbelkada for all the support as well... > > I have been trying to play around with BLIP2 and PEFT using the example notebook (https://colab.research.google.com/drive/16XbIysCzgpAld7Kd9-xz-23VPWmqdWmW?usp=sharing#scrollTo=6cCVhsmJxxjH) and a few things came up and I was hoping to get your help: > > 1. When trying to finetune with "salesforce/blip2-flan-t5-xl", I got a ton of issues: > > \`\`\` > config = LoraConfig( > r=16, > lora\_alpha=32, > lora\_dropout=0.05, > bias="none", > target\_modules=\["q\_proj", "k\_proj"\]) > \`\`\` > > The q\_proj and k\_proj layers don't exist and so I used "q","v" or tried to use just the default values and it made the loss converge to 0 extremely quickly. However, the model was really just outputting gibberish so I'm likely not using the right target\_modules... How are you supposed to tweak this parameter? In general too, is there a heuristic for these such as T5 -\> q,v , OPT -\> q\_proj,k\_proj and is that different for the regular model vs BLIP2? > > \- I tried using a bigger OPT (i.e. "ybelkada/blip2-opt-2.7b-fp16-sharded" or "ybelkada/blip2-opt-2.7b-fp16-sharded") and that just made the loss train with "nan" all the time regardless of what I tried. > > > 2. Something seemed really odd in the training loop, specifically: \`outputs = model(input\_ids=input\_ids, > pixel\_values=pixel\_values, > labels=input\_ids)\` > > \- From my understanding, this would imply that we are already passing the label into the model that we want to predict as an input? > > \- I also tried to modify the notebook to go beyond just image captioning and try to train a VQA model by modifying the following: > \`\`\` > class ImageCaptioningDataset(Dataset): > def \_\_init\_\_(self, dataset, processor): > self.dataset = dataset > self.processor = processor > > def \_\_len\_\_(self): > return len(self.dataset) > > def \_\_getitem\_\_(self, idx): > item = self.dataset\[idx\] > encoding = self.processor(images=item\["image"\],text=item\['prompt'\], padding="max\_length", return\_tensors="pt") > # remove batch dimension > encoding = {k: v.squeeze() for k, v in encoding.items()} > encoding\["text"\] = item\["text"\] > return encoding > > def collate\_fn(batch): > # pad the input\_ids and attention\_mask > processed\_batch = {} > for key in batch\[0\].keys(): > if key in \["pixel\_values",'input\_ids'\]: > processed\_batch\[key\] = torch.stack(\[example\[key\] for example in batch\]) > elif key == 'text': > text\_inputs = processor.tokenizer( > \[example\["text"\] for example in batch\], padding=True, return\_tensors="pt" > ) > processed\_batch\["input\_ids\_label"\] = text\_inputs\["input\_ids"\] > processed\_batch\["attention\_mask\_label"\] = text\_inputs\["attention\_mask"\] > return processed\_batch > > input\_ids = batch.pop("input\_ids").to(device) > input\_ids\_label = batch.pop("input\_ids\_label").to(device) > pixel\_values = batch.pop("pixel\_values").to(device, torch.float16) > > outputs = model(input\_ids=input\_ids, > pixel\_values=pixel\_values, > labels=input\_ids\_label) > \`\`\` > But then it didn't really seem to converge as well as the regular image captioning despite always having the same prompt throughout my dataset... Anything I could be doing wrong? > > Thanks in advance! > Isn’t Trainer process knows which field to use for Encoder and Decoder? By default, it seems to be manual or fixed. ---
**Author:** ![LeyaLi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leyali/32/49296_2.png) [@LeyaLi](https://discuss.huggingface.co/u/LeyaLi)\ **Post date:** [June 12, 2025, 8:05am UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/5 "2025-06-12T08:05:23Z")
I ran into the same issue and was able to resolve it. It turns out to be related to the Donut model’s `MAX_TOKEN_LEN` setting. My code runs successfully when `MAX_TOKEN_LEN` is set to 128 or lower, but the bug reappears as soon as it exceeds 128. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 12, 2025, 9:19am UTC](https://discuss.huggingface.co/t/error-fine-tuning-donut-model-using-lora/157587/6 "2025-06-12T09:19:23Z")
Oh… I see. > > > What does max\_length refer to? What are the acceptable values? > Max\_length refers to the maximum number of tokens that the text decoder of Donut can generate. > The authors set the max\_length depending on the downstream task, e.g. for DocVQA it is [set](https://github.com/clovaai/donut/blob/fa5e69b19522f475e66ee6f2f918c8f5c6194722/config/train_docvqa.yaml#L9) to 128, for RVL-CDIP it is [set](https://github.com/clovaai/donut/blob/fa5e69b19522f475e66ee6f2f918c8f5c6194722/config/train_rvlcdip.yaml#L9) to 8, etc.