# Make compute\_metric / load\_metric distributed and use multiple test sets on do\_predict

**URL:** <https://discuss.huggingface.co/t/make-compute-metric-load-metric-distributed-and-use-multiple-test-sets-on-do-predict/14646>\
**Category:** Beginners\
**Created:** [February 12, 2022, 1:49am UTC](https://discuss.huggingface.co/t/make-compute-metric-load-metric-distributed-and-use-multiple-test-sets-on-do-predict/14646 "2022-02-12T01:49:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sadra](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@sadra](https://discuss.huggingface.co/u/sadra)\
**Post date:** [February 12, 2022, 1:49am UTC](https://discuss.huggingface.co/t/make-compute-metric-load-metric-distributed-and-use-multiple-test-sets-on-do-predict/14646/1 "2022-02-12T01:49:04Z")

</div>

I am using transformer [PyTorch examples](https://github.com/huggingface/transformers/blob/v4.16.2/examples/pytorch/translation/run_translation.py) to do Machine Translation. I am computing multiple metrics on large validation sets.

1. how is it possible to Integrate distributed compute\_metrics with with `seq2seqTrainer`? (with or without `predict_with_generate`( just for during `do_eval`) ) I took a look at distributed usage of `load_metric`, however not sure if it is possible to integrate it without implementing custom train/evaluation loop.

2. how is it possible to have two sets on `do_predict`?

Thanks,

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [February 14, 2022, 4:08pm UTC](https://discuss.huggingface.co/t/make-compute-metric-load-metric-distributed-and-use-multiple-test-sets-on-do-predict/14646/2 "2022-02-14T16:08:05Z")

</div>

As long as you launch your script with the `torch.distributed` launcher, the evaluation will automatically be distributed (each GPU will see a fraction of your dataset and make the predictions on it).

To pass along two datasets for predictions, you will need to tweak the example script to run the predict method on those two datasets.

---

<div class="post-metadata">

**Author:** ![anmolagarwal999](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@anmolagarwal999](https://discuss.huggingface.co/u/anmolagarwal999)\
**Post date:** [August 25, 2023, 1:11pm UTC](https://discuss.huggingface.co/t/make-compute-metric-load-metric-distributed-and-use-multiple-test-sets-on-do-predict/14646/3 "2023-08-25T13:11:50Z")

</div>

@sgugger I am using DeepSpeed ZeroStage 3 and am passing a custom compute\_metrics() to the trainer. I have 4 GPUs (devices). The compute\_metrics function is being invoked by all the devices. Moreover, all the points (let’s say there N datapoints in the eval set) in the entire eval dataset seem to be sent to the compute\_metrics of all the devices. Am I missing something here?