# Cannot launch multi-gpu training?

**URL:** <https://discuss.huggingface.co/t/cannot-launch-multi-gpu-training/54987>\
**Category:** 🤗Transformers\
**Created:** [September 14, 2023, 10:36am UTC](https://discuss.huggingface.co/t/cannot-launch-multi-gpu-training/54987 "2023-09-14T10:36:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Data-drone](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@Data-drone](https://discuss.huggingface.co/u/Data-drone)\
**Post date:** [September 14, 2023, 10:36am UTC](https://discuss.huggingface.co/t/cannot-launch-multi-gpu-training/54987/1 "2023-09-14T10:36:10Z")

</div>

I am trying to run training on multiple GPUs following this script here:

> <https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-7b/06_fine_tune_qlora.py>

single gpu is fine. When I switch to multi-gpu I got:

```auto
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! 

```

So I tried wrapping the dataloader in accelerate.prepare.  
then I get:

```auto
AttributeError: 'DataLoaderShard' object has no attribute 'map'

```

What is the correct path forward?