Can an LLM generate the test criteria or assertions for my prompt?

It can propose them, but review the list, since picking the right checks for a task is hard. Our PromptEvals dataset has 2,087 production pipeline prompts with 12,623 assertion criteria, collected from developers using our open-source tools. Open models we fine-tuned on it beat GPT-4o at generating relevant assertions by 20.93% on average.

Sources: PromptEvals: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines (NAACL 2025)

Guide: What Are LLM Evals?

More questions on this topic

All questions about using AI for data and knowledge work