Can I use a cheaper LLM for most rows without losing accuracy?
Use BARGAIN, our method for LLM-powered data processing with guarantees: a cheap model handles a row when it can be trusted, and BARGAIN decides when that is with adaptive sampling and statistical estimates, with guarantees on accuracy, precision, or recall. Across 8 datasets, BARGAIN reduced cost by up to 86% more than prior methods.
How it works. A model cascade sends each row to a cheap model first and keeps its answer only when the model's confidence is above a threshold; the other rows go to the expensive model. The hard part is the threshold: set it too low and accuracy drops, too high and you save little. BARGAIN checks the cheap model on an adaptively chosen sample of rows and uses statistical estimates to pick the threshold that saves the most while still meeting the accuracy, precision, or recall target you set.
Sources: Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees (SIGMOD 2026)
Guide: How to Process Documents with LLMs Cheaply
More questions on this topic
- Why are AI functions in SQL (Snowflake Cortex, BigQuery AI, Databricks ai_query) so slow and expensive on big tables?
- How do I extract information accurately from thousands of long documents with an LLM?
- My LLM misses items when I ask it to find every instance of something in a long document. Should I split the prompt or the document?
- Should I ask an LLM to extract dozens of fields from a long document in one prompt, or split it up?
- How do I optimize a multi-step LLM pipeline for both accuracy and cost, not just one prompt at a time?
- Is there something better than a model cascade for LLM classification over documents?
- How do I match records across two tables with an LLM without paying for every pair?
- In what order should I run several LLM filters on the same data?
- Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?
- Should I put many records in one prompt so the LLM sees context when classifying?
- How do I get consistent 1-10 scores from an LLM across many items?
- How do I run SQL queries over thousands of similar PDFs, such as reports or forms that follow one template?
- Why does RAG fail on "how many" or aggregate questions across a whole document collection?
- How do I deduplicate records with an LLM without missing too many true matches?
- How do I add a summary of each user's history and feedback to my prompts without slowing down responses?
- Can LLM agents reliably answer business questions over real company databases, not just text-to-SQL benchmarks?
- Why do text-to-SQL and data agents return wrong answers? Is it picking the wrong table?
- Should an agent use regex or an LLM to pull values like dates out of free-text columns?
- Do AI agents need a different kind of database than human analysts do?
- Why does my LLM agent run so many SQL queries for one question, and how do I cut them down?
- How do I write assertions or guardrails that catch bad LLM outputs in production?
- Can an LLM generate the test criteria or assertions for my prompt?
- How do I catch bad data before it gets into a retraining run without drowning in false alerts?
- How do I monitor an ML model in production when labels arrive late or not at all?
- My model's accuracy on a rolling window dropped. Is the model actually getting worse?
- Can I use Prometheus and Grafana to monitor ML model accuracy?
- Should I fine-tune a model or use RAG to give an LLM my company's knowledge?
- Do long-context models make RAG obsolete?
- How should I evaluate a RAG system in production?