How to process documents with LLMs cheaply
LLMs can classify, extract from, and summarize documents that no rule-based pipeline could handle. Running one over every document in a large collection is expensive, though, and the cheapest model is often not accurate enough. Our work cuts the cost in five ways, most of it with Aditya Parameswaran and Sepanta Zeighami.
Let an optimizer rewrite the pipeline
You describe the pipeline as a sequence of LLM operations, such as map, filter, and reduce, and an optimizer rewrites it into a cheaper or more accurate plan.
- DocETL (VLDB 2025), with Tristan Chambers, Tarak Shah, Aditya Parameswaran, and Eugene Wu: an agent rewrites the pipeline, for example by splitting long documents or a hard operation, and evaluates each plan. Its plans were 25 to 80% more accurate than well-engineered baselines. The code is at docetl.org.
- MOAR (VLDB 2026), led by Lindsey Linxi Wei, with Sepanta Zeighami, Yeounoh Chung, Fatma Ozcan, and Aditya Parameswaran: an optimizer for DocETL that trades off accuracy and cost. Across six workloads, it was 27% more accurate than ABACUS, and matched ABACUS’s best accuracy at 55% of its cost.
Use a cheaper model where it is good enough
A cascade answers each document with a cheap model first, and sends only the documents the cheap model is unsure about to an expensive one.
- BARGAIN (SIGMOD 2026), led by Sepanta Zeighami, with Aditya Parameswaran: a cascade with accuracy guarantees. On 8 real-world datasets, it cut cost by up to 86% more than the previous state of the art, with stronger guarantees.
- Task Cascades (SIGMOD 2026), with Sepanta Zeighami and Aditya Parameswaran: a cascade that varies the task, not only the model: a cheap model might answer an easier related question, or read only the part of the document that matters. At a 90% target accuracy on eight document tasks, it cut cost by 36% on average compared with model cascades.
Join documents without comparing every pair
A semantic join asks an LLM whether two rows match, so a naive join makes one call for every pair of rows.
- Featurized-Decomposition Join (VLDB 2026), led by Sepanta Zeighami, with Aditya Parameswaran: an LLM extracts features from each row, such as a product’s brand, and combines them into rules that rule out most pairs with cheap comparisons. It cost up to 10 times less than prior methods at the same quality guarantees.
Use the structure documents share
Many collections, such as civic agenda reports, follow one template. ZenDB (ICDE 2025), led by Yiming Lin, detects the shared section structure of each document. You query the collection with SQL, and ZenDB sends the LLM only the sections that matter.
Run it faster on GPUs
When the pipeline runs on open-weight models, the inference engine decides much of the cost. Quail, our engine for AI functions in SQL, plans the query and the LLM inference together.
Questions
- How do I extract information accurately from thousands of long documents with an LLM?
- My LLM misses items when I ask it to find every instance of something in a long document. Should I split the prompt or the document?
- Should I ask an LLM to extract dozens of fields from a long document in one prompt, or split it up?
- How do I optimize a multi-step LLM pipeline for both accuracy and cost, not just one prompt at a time?
- Can I use a cheaper LLM for most rows without losing accuracy?
- Is there something better than a model cascade for LLM classification over documents?
- How do I match records across two tables with an LLM without paying for every pair?
- In what order should I run several LLM filters on the same data?
- Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?
- Should I put many records in one prompt so the LLM sees context when classifying?
- How do I get consistent 1-10 scores from an LLM across many items?
- How do I run SQL queries over thousands of similar PDFs, such as reports or forms that follow one template?
- Why does RAG fail on "how many" or aggregate questions across a whole document collection?
- How do I deduplicate records with an LLM without missing too many true matches?
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?