How do I make LLM-as-a-judge evaluations reliable?
We recommend pairwise choices instead of 1-to-5 scores, and checking the judge against human labels. In What We've Learned From a Year of Building with LLMs, we recommend running each comparison twice with the order swapped, allowing ties, asking for reasoning first, and matching response lengths. In one example, reviewing the judge's critiques with stakeholders raised its agreement with humans from 68% to 94% over three rounds.
Sources: What We've Learned From a Year of Building with LLMs (O'Reilly Radar); What should I do when I can’t get my LLM judge to agree with human reviewers? (AI Evals FAQ, Hamel Husain and Shreya Shankar)
Guide: What Are LLM Evals?