# Flan-t5 multi gpu training issue

**URL:** <https://discuss.huggingface.co/t/flan-t5-multi-gpu-training-issue/35387>\
**Category:** 🤗Transformers\
**Created:** [April 3, 2023, 8:26am UTC](https://discuss.huggingface.co/t/flan-t5-multi-gpu-training-issue/35387 "2023-04-03T08:26:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![shreyans92dhankhar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shreyans92dhankhar/32/4909_2.png) [@shreyans92dhankhar](https://discuss.huggingface.co/u/shreyans92dhankhar)\
**Post date:** [April 3, 2023, 8:26am UTC](https://discuss.huggingface.co/t/flan-t5-multi-gpu-training-issue/35387/1 "2023-04-03T08:26:18Z")

</div>

Hi,

I am loading flan t5 xxl sharded version using “philschmid/flan-t5-xxl-sharded-fp16” for finetuning. I want to do multi-gpu training using this. After loading the model from AutoModelForSeq2SeqLM, i am setting this parameter setattr(model, ‘model\_parallel’, True) but i am getting following error while training using trainer.  
AttributeError: ‘T5Stack’ object has no attribute ‘first\_device’

Can someone help with this issue?