Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases
By: Hui Huang , Xuanxin Wu , Muyun Yang and more
Potential Business Impact:
Makes AI judges fairer and more accurate.
This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judge to non-reasoning LLMs. Our empirical analysis yields four key findings: 1) LRMs outperform non-reasoning LLMs in terms of judgment accuracy, particularly on reasoning-intensive tasks; 2) LRMs demonstrate superior instruction-following capabilities in evaluation contexts; 3) LRMs exhibit enhanced robustness against adversarial attacks targeting judgment tasks; 4) However, LRMs still exhibit strong biases in superficial quality. To improve the robustness against biases, we propose PlanJudge, an evaluation strategy that prompts the model to generate an explicit evaluation plan before execution. Despite its simplicity, our experiments demonstrate that PlanJudge significantly mitigates biases in both LRMs and standard LLMs.
Similar Papers
Assessing Judging Bias in Large Reasoning Models: An Empirical Study
Computers and Society
Makes AI judges fairer and more trustworthy.
JudgeLRM: Large Reasoning Models as a Judge
Computation and Language
Makes AI judges better at thinking hard problems.
Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
Artificial Intelligence
Computers that "think" judge better than those that don't.