# Run\_summarization.py Rouge in eval cf. in final eval, predict

**URL:** <https://discuss.huggingface.co/t/run-summarization-py-rouge-in-eval-cf-in-final-eval-predict/9802>\
**Category:** Beginners\
**Created:** [September 8, 2021, 9:44am UTC](https://discuss.huggingface.co/t/run-summarization-py-rouge-in-eval-cf-in-final-eval-predict/9802 "2021-09-08T09:44:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheLongSentance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thelongsentance/32/2470_2.png) [@TheLongSentance](https://discuss.huggingface.co/u/TheLongSentance)\
**Post date:** [September 8, 2021, 9:44am UTC](https://discuss.huggingface.co/t/run-summarization-py-rouge-in-eval-cf-in-final-eval-predict/9802/1 "2021-09-08T09:44:40Z")

</div>

Hi,

I am using the HuggingFace run\_summarization.py example to train T5 for summarisation (and some other tasks too). I noticed that the Rouge score numbers for evaluation inside training are a lot lower than in evaluation and prediction scores at the end of training. The scores suddenly jump up.

To try to remove anything specific to my model or data, and to keep things simple, I tried running a very small example with the following parameterisation:

–model\_name\_or\_path t5-small --do\_train True --num\_train\_epochs 1.0 --max\_train\_samples 1000 --do\_eval True --evaluation\_strategy steps --eval\_steps 100 --max\_eval\_samples 100 --do\_predict True --predict\_with\_generate True --max\_predict\_samples 100 --dataset\_name cnn\_dailymail --dataset\_config “3.0.0” --source\_prefix "summarize: " --output\_dir “d:/BrianS/models/sum\_test” --per\_device\_train\_batch\_size=4 --per\_device\_eval\_batch\_size=4 --overwrite\_output\_dir --save\_steps 100 --logging\_strategy steps --logging\_steps 50

\*\*\* At the first evaluation I get the following results: \*\*\*

{‘eval\_loss’: 2.1008858680725098, ‘eval\_rouge1’: 23.2084, ‘eval\_rouge2’: 7.5598, ‘eval\_rougeL’: 18.7416, ‘eval\_rougeLsum’: 20.9243, ‘eval\_gen\_len’: 19.0, ‘eval\_runtime’: 5.5309, ‘eval\_samples\_per\_second’: 18.08, ‘eval\_steps\_per\_second’: 4.52, ‘epoch’: 0.4}

\*\*\* At the second evaluation I get: \*\*\*

{‘eval\_loss’: 2.093972682952881, ‘eval\_rouge1’: 23.4257, ‘eval\_rouge2’: 8.0721, ‘eval\_rougeL’: 19.0282, ‘eval\_rougeLsum’: 21.3982, ‘eval\_gen\_len’: 19.0, ‘eval\_runtime’: 5.4218, ‘eval\_samples\_per\_second’: 18.444, ‘eval\_steps\_per\_second’: 4.611, ‘epoch’: 0.8}

\*\*\* But once training is complete I get for evaluation: \*\*\*

epoch = 1.0  
eval\_gen\_len = 60.13  
eval\_loss = 2.0929  
eval\_rouge1 = 30.7996  
eval\_rouge2 = 11.3268  
eval\_rougeL = 23.027  
eval\_rougeLsum = 28.309  
eval\_runtime = 0:00:16.26  
eval\_samples = 100  
eval\_samples\_per\_second = 6.148  
eval\_steps\_per\_second = 1.537

\*\*\* and for prediction: \*\*\*

predict\_gen\_len = 60.37  
predict\_loss = 2.0777  
predict\_rouge1 = 29.7574  
predict\_rouge2 = 9.4481  
predict\_rougeL = 21.0967  
predict\_rougeLsum = 26.475  
predict\_runtime = 0:00:16.84  
predict\_samples = 100  
predict\_samples\_per\_second = 5.937  
predict\_steps\_per\_second = 1.484

So I noticed that \_gen\_len seemed very different between the two sets of results, plus the eval run at the end was taking so much longer than the evals during training. So I debugged inside the compute\_metrics function for the 3 evaluation runs (the two inside training, the one after training) and found:

\*\*\* The target summarisation for the first sample is: \*\*\*

_‘Accident happens in Santa Ynez, California, near where Crosby lives. The jogger suffered multiple fractures; his injuries are not believed to be life-threatening.’_

\*\*\* prediction for first eval (0.4 epoch as above) \*\*\*

_‘David Crosby hit a jogger with his car in Santa Ynez,’_

\*\*\* prediction for second eval (0.8 epoch as above) \*\*\*

_‘David Crosby was driving at approximately 50 mph when he struck the jogger’_

\*\*\* prediction for eval after training completed \*\*\*

_‘David Crosby hit a jogger with his car in Santa Ynez, California. The jogger suffered multiple fractures and was airlifted to a hospital. Crosby is known for weaving multilayered harmonies over sweet melodies.’_

So given the lengh of the prediction in the evaluation after trianing, it is of similar length to the target and it is perhaps unsurprising the scores are better. It looks like the output generated is being kept small to keep compute resources down for fast eval during training, then openned up to the full summarization at the end? Have I missed a parameter that controls this behaviour, so that you could force full generation for eval during training, not just at the end? Be good to understand this, at the moment the Rouge scores I have go up like a hockey stick at the end.

Thanks!