Where do I start with evals for my LLM app, and which metrics should I use?
Start by reading many real outputs and writing down what goes wrong; pick metrics from those failures. In Data Flywheels for LLM Applications, we recommend code checks for simple heuristics, LLM judges for subjective criteria, and binary true/false metrics, because they are easier to align and to label consistently.
Sources: Data Flywheels for LLM Applications; What’s a minimum viable evaluation setup? (AI Evals FAQ, Hamel Husain and Shreya Shankar)
Guide: What Are LLM Evals?