How do I write assertions or guardrails that catch bad LLM outputs in production?

Start from the mistakes you already fixed: each instruction you added to your prompt to stop a mistake is a candidate check. Our SPADE system does this automatically: it mines a pipeline's history of prompt edits to write candidate assertion functions, then picks a small set that meets coverage and accuracy targets. Across nine real pipelines, SPADE needed 14% fewer assertions and had 21% fewer false failures than simpler baselines. It is offered in LangSmith.

Sources: SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines (VLDB 2024)

Guide: What Are LLM Evals?

More questions on this topic

All questions about using AI for data and knowledge work