Questions about using AI for data and knowledge work
Answers to questions people ask about using AI for data analysis and knowledge work, grouped by layer of the stack, from human behavior down to inference. Each answer describes how one of our papers or posts approached the problem and links to it.
Human behavior
Where do I start with evals for my LLM app, and which metrics should I use?
Start by reading many real outputs and writing down what goes wrong; pick metrics from those failures. In Data Flywheels for LLM Applications, we recommend code checks for simple heuristics, LLM judges for subjective criteria, and binary true/false metrics, because they are easier to align and to label consistently. The FAQ entry "What's a minimum viable evaluation setup?" is a short starting point.
Sources: Data Flywheels for LLM Applications; AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar)
Can I trust an LLM-as-a-judge? How do I know it agrees with human reviewers?
Only after you check it against your own labels on a sample of real outputs. Our EvalGen paper describes a tool for this: you grade a subset of outputs, and it uses your grades to pick the candidate judge prompts and code checks that best match them. Our user study found that alignment is subjective and iterative, so keep checking as outputs change. The FAQ entry "How do I know if I can trust my automated eval?" gives a workflow.
Sources: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (UIST 2024); AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar)
Should I write my eval criteria or rubric before I look at model outputs?
Draft some, but expect to revise them once you read outputs. In our EvalGen study, we named this criteria drift: people need criteria to grade outputs, but grading outputs changes the criteria they want, and some criteria depended on the specific outputs people saw. In Data Flywheels for LLM Applications, we argue you cannot find good criteria by theorizing about failure modes alone.
Sources: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (UIST 2024); Data Flywheels for LLM Applications
Should my LLM judge give a pass/fail or a 1 to 5 score?
We recommend pass/fail as the default. As we explain in Data Flywheels for LLM Applications, binary metrics are easier to align with your judgment, because you only need to agree on what counts as true or false, and easier to label consistently over time. The FAQ entry "Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings?" goes further.
Sources: Data Flywheels for LLM Applications; AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar)
How do I make LLM-as-a-judge evaluations reliable?
We recommend pairwise choices instead of 1-to-5 scores, and checking the judge against human labels. In What We've Learned From a Year of Building with LLMs, we recommend running each comparison twice with the order swapped, allowing ties, asking for reasoning first, and matching response lengths. In one example, reviewing the judge's critiques with stakeholders raised its agreement with humans from 68% to 94% over three rounds.
Sources: What We've Learned From a Year of Building with LLMs (O'Reilly Radar)
Can I use ChatGPT or Claude to do qualitative coding or thematic analysis?
They can speed up the mechanical parts, but in our tests they did not do the analysis well. In Exploring Agent-Assisted Qualitative Analysis, we ran six agent setups for grounded theory on 451 tweets. The agents mostly paraphrased tweets instead of analyzing them, coded at most 68% of the tweets, and overfit to feedback or forgot it.
Why does LLM writing feel bad, and how do I write with an LLM without that?
The problems are mostly empty summary sentences, flat rhythm, low information density, and vagueness, not em dashes. Writing in the Age of LLMs lists these patterns and the habits people wrongly flag, such as intentional repetition and parallel structure. We recommend telling the model the story first, writing the rough paragraph yourself, and using the model for scoped rewrites.
Sources: Writing in the Age of LLMs
What is MLOps work actually like day to day?
Mostly data work, evaluation, deployment, and responding to drops in production, more than modeling. In Operationalizing Machine Learning, we interviewed 18 ML engineers across areas such as chatbots, autonomous vehicles, and finance; three things governed success: velocity, validation, and versioning. A follow-up blog post describes Task MLEs, who keep specific pipelines running, and Platform MLEs, who automate shared work.
Sources: Operationalizing Machine Learning: An Interview Study (CSCW 2024); Thoughts on ML Engineering After a Year of my PhD
Should I spend more time on the model or on the data?
Usually the data. In Reflecting on a Year of Making Machine Learning Actually Useful, the biggest gains as a startup ML engineer came from adding features or redefining the train and validation splits, not from new model types. About 90% of the work was independent of the modeling library, and a shallow decision tree on the predictions helped find bugs.
Sources: Reflecting on a Year of Making Machine Learning Actually Useful
Should I retrain my model on a fixed schedule or only when drift is detected?
For most small models, we recommend frequent scheduled retraining, which is cheap and fixes natural drift, and spending alerting effort on sudden breaks from data or engineering bugs. Thoughts on ML Engineering After a Year of my PhD describes Task MLEs at small companies who retrain hourly, daily, or weekly, and whose model performance recovers when they do. Retraining many xgboost or scikit-learn models costs a few dollars at most.
Interfaces
How do I make AI-generated summaries in a dashboard or table easy to skim?
We recommend giving every item the same structure and the same words for the same things, and putting the words that set an item apart first. Our post Five Principles for Generating Readable AI Text in Bulk draws on reading research for five rules, such as starting each item with its subject or verb and using variation only to mark a real difference. It shows fixes for a Notion AI summary column and a theme dashboard Claude generated.
Sources: Five Principles for Generating Readable AI Text in Bulk
How do I get an LLM to write consistent one-line titles or summaries for many items?
We recommend fixing one pattern and stating it in the prompt: who the subject is, which tense, and which words go first. In Five Principles for Generating Readable AI Text in Bulk, we rewrite seven agent session titles from a Claude sidebar so each starts with a verb in the continuous form, such as "Measuring vLLM performance," because readers take the first noun and verb as subject and action.
Sources: Five Principles for Generating Readable AI Text in Bulk
Is there a better way to iterate on prompts when an LLM pipeline runs over many documents?
We recommend looking at outputs across many documents, noting what goes wrong, and turning those notes into prompt changes; a single example hides most failures. Our DocWrangler IDE supports this: you annotate documents and outputs in place, and an LLM turns your notes into revised prompts or suggests splitting an operation that is too hard. In our study and public deployment, users often turned open-ended operations into classifiers so they were easier to check.
Sources: Steering Semantic Data Processing with DocWrangler (UIST 2025)
Why is it so hard to find a good dataset for my project?
Keyword search sees only titles and tags, and you often can't tell whether a dataset fits your task until you inspect its rows and columns. In our survey of 89 data professionals, finding an initial dataset was still unsolved. We built DataScout to help: it suggests query rewrites, searches dataset contents, and shows relevance indicators for your task.
Sources: It Took Longer Than I Was Expecting: Why Is Dataset Search Still So Hard? (HILDA 2024 (Workshop on Human-in-the-Loop Data Analytics)); Rethinking Dataset Discovery with DataScout (UIST 2025)
My RAG answers are wrong. How do I tell whether retrieval or generation is the problem?
We recommend testing the pieces separately; for example, give the model known-good context and see whether the answer improves. In RAG Without the Lag (CHI 2026 Best Paper), we studied how 12 engineers debug RAG pipelines and built RAGGY, which pairs composable RAG building blocks with an interface for fast, interactive debugging. We found the settings that matter most often need hours of preprocessing after each change, which slows debugging.
Sources: RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines (CHI 2026)
How do I find which cells in a messy Jupyter notebook actually produced this plot or model, so I can clean it up?
Use a program slicer, which traces the code a result depends on. Static slicers such as Gather read the code without running it, so they keep extra cells. Our nbslicer tracks dependencies while cells run; on 374 real notebook sessions, its slices were on average 66% smaller than a static slicer's and still correct, and they ran 1.34x faster.
Sources: Bolt-on, Compact, and Rapid Program Slicing for Notebooks (VLDB 2023)
When I change an earlier Jupyter cell, how do I know which later cells to rerun?
Compute a forward slice: the cells that read anything the changed cell wrote. Reactive kernels such as ipyflow, and notebooks such as marimo, do this for you. In nbslicer, we compute forward slices from dependencies seen at run time; on 374 real notebook sessions they were on average 54% smaller than a static slicer's, so fewer cells need to rerun.
Sources: Bolt-on, Compact, and Rapid Program Slicing for Notebooks (VLDB 2023)
Database systems
Why are AI functions in SQL (Snowflake Cortex, BigQuery AI, Databricks ai_query) so slow and expensive on big tables?
Each AI function calls the model once per row, and a naive join calls it once per pair of rows, so one query can make millions of calls. Quail, our open-source engine, runs AI-SQL on open models and plans the query and the inference together. Across 29 QUAIL-B queries on one H100, it was 1.84x faster on average than well-tuned vLLM baselines.
Sources: Building an Ultra-High Throughput AI-SQL Engine; What Is AI-SQL?
How do I extract information accurately from thousands of long documents with an LLM?
Use DocETL, our open-source system for LLM-powered document processing: you declare the pipeline, and LLM agents rewrite it, for example by breaking up the document or the task, and check candidate plans. On four document analysis tasks, DocETL found plans 25 to 80% more accurate than well-engineered baselines. Whatever tool you use, test the pipeline on a labeled sample.
Sources: DocETL; DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing (VLDB 2025)
My LLM misses items when I ask it to find every instance of something in a long document. Should I split the prompt or the document?
Usually split: models miss instances in long inputs even when the input fits in the context window. DocETL, our open-source system, rewrites pipelines to decompose the document or the operation; our DocETL paper uses this exact case, finding every force majeure or indemnification clause in long contracts. In MOAR, our later optimizer for DocETL, one example rewrite turns one compound filter into two simpler filters.
Sources: DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing (VLDB 2025); Multi-Objective Agentic Rewrites for Unstructured Data Processing (VLDB 2026)
Should I ask an LLM to extract dozens of fields from a long document in one prompt, or split it up?
For many unrelated fields over long documents, split the work: one long prompt with every field tends to miss some of them. In LLM-Powered Proactive Data Systems (IEEE Data Engineering Bulletin 2025), we argue the system should do this rewriting itself: break "extract fields f1 to fn" into one focused prompt per field, send the LLM only the relevant part of a long document, and leave arithmetic such as averages to SQL.
Sources: LLM-Powered Proactive Data Systems (IEEE Data Engineering Bulletin 2025)
How do I optimize a multi-step LLM pipeline for both accuracy and cost, not just one prompt at a time?
Use MOAR, our optimizer for DocETL: it searches over whole-pipeline rewrites with over 30 rewrite rules and a multi-armed bandit, because operators interact and tuning each alone can give a worse plan. Across six workloads, MOAR reached 27% higher accuracy than the next-best optimizer, and matched that optimizer's best accuracy at 55% of its cost.
Sources: Multi-Objective Agentic Rewrites for Unstructured Data Processing (VLDB 2026)
Can I use a cheaper LLM for most rows without losing accuracy?
Use BARGAIN, our method for LLM-powered data processing with guarantees: a cheap model handles a row when it can be trusted, and BARGAIN decides when that is with adaptive sampling and statistical estimates, with guarantees on accuracy, precision, or recall. Across 8 datasets, BARGAIN reduced cost by up to 86% more than prior methods.
How it works. A model cascade sends each row to a cheap model first and keeps its answer only when the model's confidence is above a threshold; the other rows go to the expensive model. The hard part is the threshold: set it too low and accuracy drops, too high and you save little. BARGAIN checks the cheap model on an adaptively chosen sample of rows and uses statistical estimates to pick the threshold that saves the most while still meeting the accuracy, precision, or recall target you set.
Sources: Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees (SIGMOD 2026)
Is there something better than a model cascade for LLM classification over documents?
Task cascades, from our SIGMOD 2026 paper, vary more than the model: an LLM agent proposes simpler or decomposed tasks and the relevant parts of each document, and we assemble them into a cascade, optionally with an accuracy guarantee. At a 90% target accuracy over eight tasks, task cascades cut cost by 36% on average compared with model cascades.
How it works. A model cascade varies only the model. A task cascade can also vary the task: a cheap model might answer an easier related question, or read only the part of the document that matters, and settle most rows that way. An LLM agent proposes these simpler tasks and document parts, and the cascade tries them from cheapest to most expensive, sending only the rows that remain uncertain to the full task.
Sources: Task Cascades for Efficient Unstructured Data Processing (SIGMOD 2026)
How do I match records across two tables with an LLM without paying for every pair?
Use Featurized-Decomposition Join, our method for semantic joins: an LLM extracts features from each record and combines them into logical rules that drop non-matching pairs with cheap comparisons, with statistical guarantees on result quality. Embedding similarity, the usual filter, often fails to predict matches. FDJ cut cost up to 10x compared with prior methods at the same guarantees.
How it works. A semantic join asks an LLM whether each pair of records matches, so cost grows with the number of pairs. FDJ first has an LLM extract features from each record, such as a product's brand or size. It combines the features into rules, such as "brands must be equal", that rule out most non-matching pairs with cheap comparisons, and sends only the remaining pairs to the LLM. It chooses the rules so the result still meets your quality target with high probability.
Sources: Featurized-Decomposition Join: Low-Cost Semantic Joins with Guarantees (VLDB 2026)
In what order should I run several LLM filters on the same data?
Run first the filters that reject the most rows for the least cost, so later filters see fewer rows. In How to Cost Your AI-Powered Filters, we rank each later filter by its cost per row divided by the fraction it rejects, following Hellerstein and Stonebraker, and also try each filter first. For three filters over 5,000 IMDB reviews, the chosen order scored 6.87 s compared with 7.30 s for the worst candidate.
Sources: How to Cost Your AI-Powered Filters
Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?
Often the right label depends on the rest of the dataset, and a model that sees one row can't know that. For example, a ticket can be urgent next to minor issues and routine next to security bugs. In HoldUp, labeling rows one at a time averaged under 70% accuracy on 10 classification datasets. HoldUp clusters related records with the LLM and labels the clusters jointly, reaching 80% on average and up to 33% higher.
Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)
Should I put many records in one prompt so the LLM sees context when classifying?
It can save cost, but it didn't improve accuracy in our tests, because long inputs hurt LLM quality. In HoldUp, batching records into one prompt averaged 43% classification accuracy, compared with 66% one row at a time, and asking the LLM to write a rubric from a sample reached 59%. HoldUp instead gives the LLM small random samples many times, and reached 80%.
Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)
How do I get consistent 1-10 scores from an LLM across many items?
Scoring each item alone on an absolute scale drifts; comparing items gives steadier results. In HoldUp, we group similar records, ask the LLM which group should score higher, and sort the groups with an integer linear program. On 5 scoring datasets, row-by-row scoring averaged 48% accuracy and HoldUp 64%.
Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)
How do I run SQL queries over thousands of similar PDFs, such as reports or forms that follow one template?
We recommend defining a table over the documents and extracting only the fields a query needs, using the shared template to find them. Our ZenDB system (ICDE 2025) detects the common section structure of templatized documents, such as civic agenda reports, as a tree for each document. You then query the collection with SQL, and ZenDB sends the LLM only the sections that matter.
Sources: Querying Templatized Document Collections with Large Language Models (ICDE 2025)
Why does RAG fail on "how many" or aggregate questions across a whole document collection?
Retrieval returns a few chunks, but a count needs a value from every document. ZenDB answers SQL queries, including aggregations, by locating the right section in each document through its template. On three real document collections, it beat RAG baselines by up to 61% in precision and 80% in recall, at slightly higher cost.
Sources: Querying Templatized Document Collections with Large Language Models (ICDE 2025)
How do I deduplicate records with an LLM without missing too many true matches?
Asking an LLM about each pair alone tends to give high precision and low recall; using the other answers to fix mistakes helps. In Revisiting Prompt Engineering via Declarative Crowdsourcing (CIDR 2024), gpt-3.5-turbo matched citation pairs with 0.952 precision but 0.503 recall. Marking two records as duplicates when the LLM linked them through other records raised F1 by over 6%.
Sources: Revisiting Prompt Engineering via Declarative Crowdsourcing (CIDR 2024)
How do I add a summary of each user's history and feedback to my prompts without slowing down responses?
We recommend updating the summary in the background and letting requests read the latest finished version, even if it is slightly stale. Our Motion system (SIGMOD 2024 demo) treats a prompt's parts as views that are updated as new feedback arrives, under a weak consistency model. In our shopping demo, a fresh GPT-4 summary could take over thirty seconds, while Motion returned recommendations in 1.5 seconds.
Sources: Building Reactive Large Language Model Pipelines with Motion (SIGMOD 2024 (Demo))
Can LLM agents reliably answer business questions over real company databases, not just text-to-SQL benchmarks?
Not yet, once the data is spread across systems and messy. Our DAB benchmark has 104 questions over 17 datasets in PostgreSQL, MongoDB, SQLite, and DuckDB, based on a study of six enterprises building data agents. The best agent, GPT-5.5 in Codex, answered 57% correctly on the first try, and 70% within five tries.
Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)
Why do text-to-SQL and data agents return wrong answers? Is it picking the wrong table?
Usually not; agents mostly find the right data and then compute the wrong thing. In DAB, we labeled failed agent runs: wrong implementations, such as a regex that matched part of an ISBN instead of a year, caused 57% of wrong answers. Wrong plans caused 31%, and picking the wrong data only 11%.
Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)
Should an agent use regex or an LLM to pull values like dates out of free-text columns?
Regex works for a few fixed formats and breaks on varied natural language. In DAB, all three agents we tested defaulted to regex for text extraction. On the patents dataset, dates appear as "dated 5th March 2019" and "March the 18th, 2019"; no agent's regex covered them, and every agent scored 0%. A regex for MALE also matched inside FEMALE.
Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)
Do AI agents need a different kind of database than human analysts do?
Likely yes, because agents query far more often and far more speculatively than people. In Supporting Our AI Overlords (CIDR 2026), we call this agentic speculation: many exploratory, often redundant requests on the way to an answer. We argue for databases that accept "probes" with natural-language goals, share work across them, answer approximately when that is enough, and steer agents with hints.
Sources: Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First (CIDR 2026)
Why does my LLM agent run so many SQL queries for one question, and how do I cut them down?
Agents explore schemas and try partial queries before the final one, and more attempts do raise accuracy. In Supporting Our AI Overlords (CIDR 2026), more attempts raised success on the BIRD text-to-SQL benchmark by 14% to 70%, but across 50 attempts per task, distinct sub-plans were often under 10-20% of the total, so much of the work repeats. We recommend giving the agent grounding hints, such as which column holds the needed data; that cut queries by more than 20%.
Sources: Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First (CIDR 2026)
How do I write assertions or guardrails that catch bad LLM outputs in production?
Start from the mistakes you already fixed: each instruction you added to your prompt to stop a mistake is a candidate check. Our SPADE system does this automatically: it mines a pipeline's history of prompt edits to write candidate assertion functions, then picks a small set that meets coverage and accuracy targets. Across nine real pipelines, SPADE needed 14% fewer assertions and had 21% fewer false failures than simpler baselines. It is offered in LangSmith.
Sources: SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines (VLDB 2024)
Can an LLM generate the test criteria or assertions for my prompt?
It can propose them, but review the list, since picking the right checks for a task is hard. Our PromptEvals dataset has 2,087 production pipeline prompts with 12,623 assertion criteria, collected from developers using our open-source tools. Open models we fine-tuned on it beat GPT-4o at generating relevant assertions by 20.93% on average.
How do I catch bad data before it gets into a retraining run without drowning in false alerts?
We recommend validating each new batch or partition against recent history and tuning the checks for precision, since per-feature thresholds catch most problems but alert too often. Our GATE system summarizes each data partition with quality metrics and flags corrupted partitions before retraining. At a large tech company, GATE gave a 2.1x average precision gain over prior work.
Sources: Automatic and Precise Data Validation for Machine Learning (CIKM 2023)
How do I monitor an ML model in production when labels arrive late or not at all?
Track inputs and predictions right away, and compute accuracy-style metrics as labels arrive, knowing each window is incomplete until then. In our position paper Rethinking Streaming Machine Learning Evaluation, we argue that tracking accuracy alone on unbounded, time-ordered data misses problems, partly because labels arrive late, and we recommend additional metrics. The Modern ML Monitoring Mess series works through why.
Sources: Rethinking Streaming Machine Learning Evaluation (ICLR 2022: Workshop on ML Evaluation Standards); The Modern ML Monitoring Mess: Rethinking Streaming Evaluation (1/4)
My model's accuracy on a rolling window dropped. Is the model actually getting worse?
Not necessarily: a windowed metric can move even when the model's predictive power hasn't changed. The Modern ML Monitoring Mess (part 1) gives three causes: class ratios differ across windows, windows hold different numbers of examples (fewer requests on a Sunday than a Monday), and labels arrive late, so recent predictions lack labels. We recommend checking these before triggering a retrain.
Sources: The Modern ML Monitoring Mess: Rethinking Streaming Evaluation (1/4)
Can I use Prometheus and Grafana to monitor ML model accuracy?
They work for latency, errors, and input statistics, but poorly for metrics like accuracy that join predictions with later labels. In The Modern ML Monitoring Mess (part 3), we tried to monitor cumulative accuracy for a toy taxi-tip model with Prometheus. It needed high-cardinality joins and convoluted PromQL, results depended on the scrape interval, and accuracy couldn't be computed over sliding windows.
Sources: The Modern ML Monitoring Mess: Failure Modes in Extending Prometheus (3/4)
Should I fine-tune a model or use RAG to give an LLM my company's knowledge?
We recommend starting with RAG for new knowledge, and fine-tuning only after you have shown that prompting and retrieval fall short. In What We've Learned From a Year of Building with LLMs, we cite studies where RAG beat fine-tuning on both known and new facts. Retrieval indexes are also cheaper to update, easy to purge of bad documents, and can be split so each customer sees only its own data.
Sources: What We've Learned From a Year of Building with LLMs (O'Reilly Radar)
Do long-context models make RAG obsolete?
No. Even with a 10M-token window you still have to choose what goes in it. In What We've Learned From a Year of Building with LLMs (O'Reilly Radar), we note there was little evidence that models reason well over very long inputs beyond needle-in-a-haystack tests. Irrelevant context can distract the model, and inference time grows with context length, so reading everything for every question costs more.
Sources: What We've Learned From a Year of Building with LLMs (O'Reilly Radar)
How should I evaluate a RAG system in production?
We recommend evaluating retrieval and generation separately, using failures you find in real traces; the evals FAQ entry "How should I approach evaluating my RAG system?" covers this. For pipelines with several LLM steps, Data Flywheels for LLM Applications recommends placing validators at each step and computing per-step accuracy, so you can see where errors compound.
Sources: AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar); Data Flywheels for LLM Applications
Inference
What is the fastest, cheapest way to run an LLM over every row of a large dataset?
Use Quail, our open-source engine for AI functions in SQL: it runs an open model as an offline batch job, keeps the GPU full, and plans the query and the inference together to reuse work across rows. Two AI filters over all 100,000 IMDB reviews cost $0.37 on one H100 at Modal's price, including model startup. The same workload at GPT-5 nano prices would cost about $1.75.
My vLLM batch job has thousands of requests but the GPU is still underused. Why?
Often the CPU is the bottleneck: scheduling and tracking millions of short requests leaves gaps between GPU kernels. Recomputing evicted KV adds more work. On our BIO-4 query, stock vLLM took 6.84 hours, 27.55x our speed-of-light estimate, and computed 40% more tokens than needed. Quail, which plans KV reuse from the query and fills each forward pass, took 29.26 minutes.
How it works. Before each forward pass, the CPU decides which requests run and prepares their inputs. With millions of short requests, that CPU work can take longer than the GPU's, so the GPU sits idle between passes; a profiler trace shows these as gaps. A second cost is what we call KV regret: the engine discards KV it needs again later and recomputes it. Quail reduces both by planning its batches from the query.
When does speculative decoding actually help, and does it still help at large batch sizes?
Speculative decoding helps when decoding is memory-bound and the drafts are cheap and usually right; at large batches the extra verify tokens cost real compute, so the draft length matters. In our AI.EXTRACT work in Quail, we draft by copying tokens from the document and pick a draft length for each request as the query runs. The best fixed length was 4 on some queries and 32 on others, and a wrong fixed length cost up to 20%.
How it works. An LLM normally writes one token per forward pass, and each pass reads all the model weights and the document's KV from GPU memory, so decoding is limited by memory reads, not arithmetic. Speculative decoding guesses the next few tokens, called the draft, and checks them all in one forward pass; if the model agrees, you get several tokens for the cost of one. Long drafts pay off when they're usually right; when they're often wrong, the extra checked tokens are wasted work. Quail learns, as the query runs, how often drafts are accepted and how long answers are, and picks a draft length for each request.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document
Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
It works best when the output copies the input, as in extraction, and helps little for freely written text. In Quail's AI.EXTRACT, every answer is a substring of the document, so we copy the draft from the first match in the document and learn, per field, how often drafts are accepted and how long answers run. Quail was 1.8x faster on average than vLLM with n-gram speculation and sorted prompts, and 4.7x faster than naive vLLM.
How it works. Prompt lookup decoding drafts tokens by finding the text written so far in the prompt and guessing that what followed it there comes next. In extraction, every answer is copied from the document, so these guesses are usually right. Quail also constrains decoding so the answer must stay a substring of the document, and it sets the draft length for each request from what it has learned about that field.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document
How do I reuse the KV cache when I ask many questions about the same long document?
Quail handles this for you: it plans KV reuse from the query and keeps a document's KV only while a later operator will read it. On AGENT-5, which runs three classifications over the same traces, 64% of the tokens vLLM computed were recomputations. With a general engine such as vLLM, put the document first and the question last, send a document's questions together, and turn on prefix caching; an LRU cache can still evict the document before the next question arrives.
How it works. When an LLM reads a document, it stores intermediate results for every token, called the KV cache, so it doesn't redo that work when it writes the answer. If you ask a second question about the same document, the engine can reuse that KV instead of reading the document again, but only if it's still in GPU memory. Quail knows from the query which operators will read each document, so it keeps the KV exactly until the last one runs.
Sources: To Classify, Try a Trie Tree; Building an Ultra-High Throughput AI-SQL Engine
Does the order of my requests matter for vLLM prefix caching in an offline batch job?
Yes. Quail plans KV reuse from the query, so you don't order requests by hand. In our AI.EXTRACT experiments, naive vLLM sent each document's questions in random order and took 2,079 s on CUAD-16, which asks 41 questions of each contract. If you use vLLM directly, sort the requests so a document's questions sit together; in the same experiments, that removed most of the waste.
How it works. Prefix caching keeps the KV of recently seen prompt prefixes in GPU memory, so a later request with the same prefix, such as the same document, can skip recomputing it. Memory is limited, so the cache evicts entries, usually the least recently used. If a document's questions are spread across the batch, its KV is often evicted before the next question arrives and must be computed again. Quail plans KV reuse from the query, so it knows which requests share a document.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document; Building an Ultra-High Throughput AI-SQL Engine
How do I force an LLM to answer with only one of my class labels?
Write the query with Quail's AI.CLASSIFY, which restricts the model to your labels: it builds a trie of the label tokens and runs the model only where labels branch apart. With 10 support-ticket labels, that averages 1.5 decode requests per document, and on 12 classification queries Quail was 1.8x faster per query than vLLM generating and parsing the label. Without Quail, you can use structured or constrained decoding in vLLM or an LLM API to limit the output to your labels.
How it works. A trie, or prefix tree, stores each label as a path of tokens; labels that start with the same words share the start of their path. Quail only asks the model to choose where paths branch apart. For example, with the labels "shipping delay", "shipping damage", and "shipping address change", once the model picks "shipping" and then "address", the word "change" is already determined, so Quail skips that decode step. Constrained decoding in vLLM or an API also limits output to valid labels, but it typically still decodes every token.
Sources: To Classify, Try a Trie Tree
How do I get a probability for each label from an LLM classifier, when labels are more than one token?
Quail's AI.CLASSIFY can return a probability for every label. Its trie_tree method scores each label only up to the word where it branches away from the other labels, then normalizes over all labels, so long labels aren't penalized. Quail also has a method that puts a letter before each label, so each label is one token, and it picks the cheaper method for your model and labels.
How it works. A label like "refund request" spans several tokens, and the model gives a probability for each one. Multiplying all of them gives longer labels lower scores even when the model favors them. trie_tree scores each label only up to the point where it splits from the other labels, since later tokens add no information, then divides by the sum over all labels so the probabilities add up to 1.
Sources: To Classify, Try a Trie Tree
How can I estimate how long LLM inference over a dataset will take on a given GPU?
Our post How to Cost Your AI-Powered Filters works through a roofline estimate: count the FLOPs and the bytes moved, then divide by the GPU's peak compute and memory bandwidth. For Qwen3-4B on an H100, one true/false filter over 5,000 IMDB reviews comes out to 6.64 seconds, and a forward pass needs only about 296 tokens to become compute-bound. Quail uses these estimates in its cost model.
How it works. Every step of a forward pass does arithmetic and moves bytes between GPU memory and the compute units. The roofline model estimates a step's time as the slower of the two: arithmetic divided by peak compute, or bytes divided by memory bandwidth. A step is compute-bound when the arithmetic is slower and memory-bound otherwise. Processing more tokens per forward pass reuses the same weights for more work, which is why about 296 tokens per pass is enough to pass the H100's FP8 ridge point.
Sources: How to Cost Your AI-Powered Filters
Models
Do adversarial examples that fool neural networks also fool humans?
Under some conditions, yes. In Adversarial Examples that Fool both Computer Vision and Time-Limited Humans (NIPS 2018), we made image perturbations that transfer across an ensemble of 10 ImageNet models, each with a layer that mimics the retina. We showed the images to 38 people for about 70 ms. The perturbations pushed their answers toward the targeted wrong class more often than chance.
Sources: Adversarial examples that fool both computer vision and time-limited humans (NIPS 2018)
Are ImageNet and Open Images geographically biased, and does it affect model accuracy outside the US and Europe?
Yes. In No Classification without Representation (NIPS 2017 workshop on ML for the developing world), we geolocated images in both datasets. About 45% of the ImageNet sample and over 32% of the Open Images sample came from the US, while China and India together held about 3%. Models trained on them scored bridegroom photos from Ethiopia and Pakistan less consistently than ones from the US and Australia.