Should my LLM judge give a pass/fail or a 1 to 5 score?
We recommend pass/fail as the default. As we explain in Data Flywheels for LLM Applications, binary metrics are easier to align with your judgment, because you only need to agree on what counts as true or false, and easier to label consistently over time.
Sources: Data Flywheels for LLM Applications; Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)? (AI Evals FAQ, Hamel Husain and Shreya Shankar)
Guide: What Are LLM Evals?