Is there a better way to iterate on prompts when an LLM pipeline runs over many documents?
We recommend looking at outputs across many documents, noting what goes wrong, and turning those notes into prompt changes; a single example hides most failures. Our DocWrangler IDE supports this: you annotate documents and outputs in place, and an LLM turns your notes into revised prompts or suggests splitting an operation that is too hard. In our study and public deployment, users often turned open-ended operations into classifiers so they were easier to check.
Sources: Steering Semantic Data Processing with DocWrangler (UIST 2025)
Guide: Interfaces for Data Work with LLMs
More questions on this topic
- How do I make AI-generated summaries in a dashboard or table easy to skim?
- How do I get an LLM to write consistent one-line titles or summaries for many items?
- Why is it so hard to find a good dataset for my project?
- My RAG answers are wrong. How do I tell whether retrieval or generation is the problem?
- How do I find which cells in a messy Jupyter notebook actually produced this plot or model, so I can clean it up?
- When I change an earlier Jupyter cell, how do I know which later cells to rerun?