Is there a better way to iterate on prompts when an LLM pipeline runs over many documents?

We recommend looking at outputs across many documents, noting what goes wrong, and turning those notes into prompt changes; a single example hides most failures. Our DocWrangler IDE supports this: you annotate documents and outputs in place, and an LLM turns your notes into revised prompts or suggests splitting an operation that is too hard. In our study and public deployment, users often turned open-ended operations into classifiers so they were easier to check.

Sources: Steering Semantic Data Processing with DocWrangler (UIST 2025)

Guide: Interfaces for Data Work with LLMs

More questions on this topic

All questions about using AI for data and knowledge work