# How can I get advantage using multi-GPUs

**URL:** <https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305>\
**Category:** Beginners\
**Created:** [January 20, 2021, 2:05pm UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305 "2021-01-20T14:05:38Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![HyeyeonKoo](https://avatars.discourse-cdn.com/v4/letter/h/c2a13f/32.png) [@HyeyeonKoo](https://discuss.huggingface.co/u/HyeyeonKoo)\
**Post date:** [January 20, 2021, 2:05pm UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/1 "2021-01-20T14:05:38Z")

</div>

Hello.  
I try to train RoBERTa from scratch. I have several V100 GPUs. I already know that huggingface’s transformers automatically detect multi-gpu. And I checked it for myself in training log. But, there is something I couldn’t understand. There is no improvement performance between using single and multi GPUs. I experimented 3 cases, which are training same model with same batch size on single-GPU, 2-GPUs, 4-GPUs. There are code and training loss graph for comparing below.

```auto
from transformers import RobertaConfig
config = RobertaConfig(
    num_hidden_layers=4,    
    hidden_size=512,    
    hidden_dropout_prob=0.1,
    num_attention_heads=8,
    attention_probs_dropout_prob=0.1,    
    intermediate_size=2048,    
    vocab_size=34492,
    type_vocab_size=1,    
    initializer_range=0.02,
    max_position_embeddings=512,
    position_embedding_type="absolute"
)

from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("tokenizer", max_len=512)

from transformers import RobertaForMaskedLM
model = RobertaForMaskedLM(config=config)

from transformers import LineByLineTextDataset
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="train.txt",
    block_size=tokenizer.max_len_single_sentence
)

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments
num_train_epochs = 4
max_steps = num_train_epochs * len(train_dataset)
warmup_steps = int(max_steps*0.05)

training_args = TrainingArguments(
    output_dir="output",
    overwrite_output_dir=True,
    
    do_train=True,
    max_steps=max_steps,
    warmup_steps=warmup_steps,
    num_train_epochs=num_train_epochs,

    per_device_train_batch_size=100,
    
    learning_rate=5e-5,
    
    weight_decay=0,
    max_grad_norm=1,
    
    adam_beta1=0.9,
    adam_beta2=0.98,
    adam_epsilon=1e-6,
    
# disable_tqdm=True
    logging_dir="log",
    logging_first_step=True
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
)

trainer.train()

```

 ![KakaoTalk_20210120_224037741](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e90f007459a83bf295bd1dc6dcd3513680f1070e.jpeg)  
The dark blue line is using 4-GPUs, grey line is using 2-GPUs and sky blue line is using single-GPU. As the number of GPU increases, the number of steps(x-axis) are much smaller. I understand that the shape of the loss reduction is the same. However I couldn’t understand why multi-GPU’s training speed is more slower than single-GPU. If it is normal, How can I upgrade performance with multi-GPU in this code? Is there option I can tune?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [January 22, 2021, 7:02am UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/2 "2021-01-22T07:02:58Z")

</div>

Hi @HyeyeonKoo

for multi-GPU training you need to launch the script using `torch.distributed.launch`, have a look at this section of example docs

> **[transformers/examples at main · huggingface/transformers](https://github.com/huggingface/transformers/tree/main/examples#distributed-training-and-mixed-precision)**
>
> main/examples

---

<div class="post-metadata">

**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\
**Post date:** [January 29, 2021, 10:56pm UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/3 "2021-01-29T22:56:51Z")

</div>

@valhalla Is it possible to use multiple GPU without using Trainer? If so, are there some examples somewhere by any chance? Thanks.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [January 30, 2021, 11:14am UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/4 "2021-01-30T11:14:48Z")

</div>

Hi @thecity2, you can have a look at the scripts in the `examples/legacy` folder here: [transformers/examples/legacy at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples/legacy)

For example, the `run_squad.py` script shows you how this is done: [transformers/run\_squad.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/examples/legacy/question-answering/run_squad.py)

---

<div class="post-metadata">

**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\
**Post date:** [February 3, 2021, 8:49pm UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/5 "2021-02-03T20:49:43Z")

</div>

From what I can see, there are only a couple changes for doing multi-gpu training:

```
if args.n_gpu > 1:
        model = torch.nn.DataParallel(model)

if args.n_gpu > 1:
                loss = loss.mean() # mean() to average on multi-gpu parallel (not distributed) training

```

Seems fairly straightforward unless I’m missing something else.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [February 3, 2021, 10:34pm UTC](https://discuss.huggingface.co/t/how-can-i-get-advantage-using-multi-gpus/3305/6 "2021-02-03T22:34:53Z")

</div>

You’d probably also need to use distributed barriers in your code for things like downloading a pretrained model only once. You may also need to gather the results to a single process and do some processing or evaluation. I advise you to go through the code for a better understanding. But there is not one specific implementation necessary for transformers-specific multi-GPUs. Yoy can just use transformers models like any other pytorch/tensorflow model.