Where do I start with evals for my LLM app, and which metrics should I use?

Start by reading many real outputs and writing down what goes wrong; pick metrics from those failures. In Data Flywheels for LLM Applications, we recommend code checks for simple heuristics, LLM judges for subjective criteria, and binary true/false metrics, because they are easier to align and to label consistently.

Sources: Data Flywheels for LLM Applications; What’s a minimum viable evaluation setup? (AI Evals FAQ, Hamel Husain and Shreya Shankar)

Guide: What Are LLM Evals?

More questions on this topic

All questions about using AI for data and knowledge work