# Get multiple metrics when using the huggingface trainer **URL:** \ **Category:** 🤗Transformers\ **Created:** [March 17, 2022, 5:53pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817 "2022-03-17T17:53:15Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hawoihgawjlj](https://avatars.discourse-cdn.com/v4/letter/h/c6cbf5/32.png) [@hawoihgawjlj](https://discuss.huggingface.co/u/hawoihgawjlj)\ **Post date:** [March 17, 2022, 5:53pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/1 "2022-03-17T17:53:15Z")
Hi all, I’d like to ask if there is any way to get multiple metrics during fine-tuning a model. Now I’m training a model for performing the GLUE-STS task, so I’ve been trying to get the pearsonr and f1score as the evaluation metrics. I referred to the link ([Log multiple metrics while training](https://discuss.huggingface.co/t/log-multiple-metrics-while-training/8115)) in order to achieve it, but in the middle of the second training epoch, it gave me the following error: ```auto Trainer is attempting to log a value of "{'pearsonr': 0.8609849499038021}" of type for key "eval/pearsonr" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute. Trainer is attempting to log a value of "{'f1': 0.8307692307692308}" of type for key "eval/f1" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute. --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 trainer.train() 2 frames /usr/local/lib/python3.7/dist-packages/transformers/trainer.py in _save_checkpoint(self, model, trial, metrics) 1724 self.state.best_metric is None 1725 or self.state.best_model_checkpoint is None -> 1726 or operator(metric_value, self.state.best_metric) 1727 ): 1728 self.state.best_metric = metric_value TypeError: '>' not supported between instances of 'dict' and 'dict' ``` And this is my compute\_metrics code snippet: ```auto def compute_metrics(eval_pred): predictions, labels = eval_pred predictions = predictions[:, 0] binary_predictions = [1.0 if prediction >= 3.0 else 0.0 for prediction in predictions] binary_labels = [1.0 if label >= 3.0 else 0.0 for label in labels] pr = metric_pearsonr.compute(predictions=predictions, references=labels) f1 = metric_f1.compute(predictions=binary_predictions, references=binary_labels) return {"pearsonr": pr, "f1": f1} ``` It works fine if I only use one of the metrics like `return pr` or `return f1`. Does anyone have suggestions about this issue? I’d really appreciate it. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [March 17, 2022, 6:02pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/2 "2022-03-17T18:02:04Z")
The first line in your error message indicates that it expects a scalar instead of a dictionary (`Trainer is attempting to log a value of "{'pearsonr': 0.8609849499038021}" of type for key "eval/pearsonr" as a scalar.`). Can you share why you want to return the values in a dictionary and not as values (i.e. why not use `return pr, f1`)? ---
**Author:** ![hawoihgawjlj](https://avatars.discourse-cdn.com/v4/letter/h/c6cbf5/32.png) [@hawoihgawjlj](https://discuss.huggingface.co/u/hawoihgawjlj)\ **Post date:** [March 17, 2022, 6:15pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/3 "2022-03-17T18:15:05Z")
Hi Thanks for the reply. There’s no certain reason that I used a dictionary because I just followed the way in this discussion ([Log multiple metrics while training](https://discuss.huggingface.co/t/log-multiple-metrics-while-training/8115)). I also tried `return pr, f1` as you suggested but it showed me another error: ```auto --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 trainer.train() 3 frames /usr/local/lib/python3.7/dist-packages/transformers/trainer.py in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix) 2511 2512 if all_losses is not None: -> 2513 metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item() 2514 2515 # Prefix all keys with metric_key_prefix + '_' TypeError: 'tuple' object does not support item assignment ``` ---
**Author:** ![lucadini](https://avatars.discourse-cdn.com/v4/letter/l/d9b06d/32.png) [@lucadini](https://discuss.huggingface.co/u/lucadini)\ **Post date:** [March 18, 2022, 10:47am UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/4 "2022-03-18T10:47:07Z")
I don’t know if this can help you in some way. I have implemented more metrics but not using the trainer by modifying the evaluation cycle like this: ```auto accuracy = load_metric("accuracy") precision = load_metric("precision") recall = load_metric("recall") f1 = load_metric("f1") metrics = [accuracy, precision, recall, f1] model.eval() for step, batch in enumerate(eval_dataloader): outputs = model(**batch) predictions = outputs.logits.argmax(dim=-1) if not is_regression else outputs.logits.squeeze() for metric in metrics: metric.add_batch( predictions=accelerator.gather(predictions), references=accelerator.gather(batch["labels"]), ) logger.info(f"epoch {epoch+1}: train loss {loss}") for metric in metrics: if metric.name == "accuracy": eval_metric = metric.compute() logger.info(f"{eval_metric}") else: eval_metric = metric.compute(average=None) logger.info(f"{eval_metric}") if metric.name == "f1": avg_f1 = sum(values)/2 logger.info(f"Average f1: {avg_f1}") ``` ---
**Author:** ![rowan1224](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rowan1224/32/12627_2.png) [@rowan1224](https://discuss.huggingface.co/u/rowan1224)\ **Post date:** [November 19, 2022, 11:35pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/5 "2022-11-19T23:35:29Z")
I know this is already old. The reason for this error is that `pr` and `f1` both are dict. To get multiple metrics for the log, it should be `{"pearsonr": pr['pearsonr'], "f1": f1['f1']}`. ---
**Author:** ![zayuki](https://avatars.discourse-cdn.com/v4/letter/z/43a26b/32.png) [@zayuki](https://discuss.huggingface.co/u/zayuki)\ **Post date:** [September 6, 2023, 3:39pm UTC](https://discuss.huggingface.co/t/get-multiple-metrics-when-using-the-huggingface-trainer/15817/6 "2023-09-06T15:39:34Z")
Thanks! Your reply helps a lot!