BoN Appetit Team at LeWiDi-2025: Best-of-N Test-time Scaling Can Not Stomach Annotation Disagreements (Yet)
By: Tomas Ruiz , Siyao Peng , Barbara Plank and more
Potential Business Impact:
Makes AI better at understanding disagreements.
Test-time scaling is a family of techniques to improve LLM outputs at inference time by performing extra computation. To the best of our knowledge, test-time scaling has been limited to domains with verifiably correct answers, like mathematics and coding. We transfer test-time scaling to the LeWiDi-2025 tasks to evaluate annotation disagreements. We experiment with three test-time scaling methods: two benchmark algorithms (Model Averaging and Majority Voting), and a Best-of-N sampling method. The two benchmark methods improve LLM performance consistently on the LeWiDi tasks, but the Best-of-N method does not. Our experiments suggest that the Best-of-N method does not currently transfer from mathematics to LeWiDi tasks, and we analyze potential reasons for this gap.
Similar Papers
Efficient Test-Time Scaling via Self-Calibration
Machine Learning (CS)
Makes AI smarter by guessing answers better.
Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding
Computation and Language
Makes AI smarter and faster without extra work.
Investigating Test-Time Scaling with Reranking for Machine Translation
Computation and Language
Makes computer translations better by trying many options.