Post
52
Is Jev a Good Multilingual Judge?
This paper evaluated Jev against generative LLM judges on four multilingual benchmarks
- covering up to 47 language varieties and
- three evaluation formats: binary decisions, pairwise comparisons, and graded ratings.
Key Insights
[1] At approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification.
[2] Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks.
Paper - https://arxiv.org/abs/2611.14861
This paper evaluated Jev against generative LLM judges on four multilingual benchmarks
- covering up to 47 language varieties and
- three evaluation formats: binary decisions, pairwise comparisons, and graded ratings.
Key Insights
[1] At approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification.
[2] Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks.
Paper - https://arxiv.org/abs/2611.14861