How to process documents with LLMs cheaply

LLMs can classify, extract from, and summarize documents that no rule-based pipeline could handle. Running one over every document in a large collection is expensive, though, and the cheapest model is often not accurate enough. Our work cuts the cost in five ways, most of it with Aditya Parameswaran and Sepanta Zeighami.

Let an optimizer rewrite the pipeline

You describe the pipeline as a sequence of LLM operations, such as map, filter, and reduce, and an optimizer rewrites it into a cheaper or more accurate plan.

Use a cheaper model where it is good enough

A cascade answers each document with a cheap model first, and sends only the documents the cheap model is unsure about to an expensive one.

Join documents without comparing every pair

A semantic join asks an LLM whether two rows match, so a naive join makes one call for every pair of rows.

Use the structure documents share

Many collections, such as civic agenda reports, follow one template. ZenDB (ICDE 2025), led by Yiming Lin, detects the shared section structure of each document. You query the collection with SQL, and ZenDB sends the LLM only the sections that matter.

Run it faster on GPUs

When the pipeline runs on open-weight models, the inference engine decides much of the cost. Quail, our engine for AI functions in SQL, plans the query and the LLM inference together.

Questions