Back to blog

Copy That? Faster AI-Powered Extraction by Drafting from the Document

Speedup over naive vLLM and average cost per query on four QUAIL-B queries. Naive vLLM: 1.0x and $0.69 per query. Tuned vLLM: 2.6x and $0.13. Quail: 4.7x and $0.07.
Figure 1. Speedup over naive vLLM (geometric mean, left) and average cost per query on an H100 at $3.95 per hour (right), on four QUAIL-B queries that ask several questions of each document. Section 5 describes the baselines and the queries.

1. Introduction

Extraction is one of the most common AI operations over data: given a document and a question, return the part of the document that answers it. For example, we might want to, from contracts, extract the governing law; from clinical notes, extract drug doses; from agent traces, extract signs of reward hacking (e.g., commands that peek at the grader code).

We implemented extraction as the function AI.EXTRACT in Quail, our open-source SQL engine for AI functions. This post explains how Quail runs AI.EXTRACT as fast as possible on NVIDIA GPUs.

We’ll cover:

2. AI.EXTRACT in Quail

AI.EXTRACT is a SQL function that takes in a document and a set of questions, and returns a struct with the answers, one field for each question. Each answer is a substring of the document. If the document does not answer a question, the field is NULL. AI.EXTRACT can also return the start and end of each answer (i.e., the span) in the document.

For example, suppose that a table messages has one row for each message generated by a coding agent, such as the thoughts of the agent, its commands, and their output. Suppose we want to find messages pertaining to reward hacking, i.e., messages where the agent cheats instead of solving the task, e.g., by peeking at the tests. The following query extracts the plan to hack or cheat and the command that cheats:

SELECT m.id,
       AI.EXTRACT(m.text, {
         'plan': 'Which thought indicates that the agent plans to cheat?',
         'command': 'Which command peeks at the tests or their answers?'
       }) AS cheat
FROM messages m;

3. Extracting with an Off-the-Shelf LLM

Suppose that we want to run AI.EXTRACT with an off-the-shelf LLM (i.e., any autoregressive LLM, not a model trained specifically for extraction). The simplest way is to send the LLM one request for each document and field, and get one answer for each request, as in Figure 2.

One AI.EXTRACT request: a message of a coding agent, the question of the command field, the instruction, and the answer cat /tests/*, copied from the message.
Figure 2. One request of the query in Section 2, for the command field, on one message of a coding agent from Terminal Wrench (shortened). After Phrase:, the LLM writes the answer cat /tests/*, copied from the commands in the message.

The answer must be a substring of the document (which trivially includes the empty string). However, an LLM can write any text. So, Quail constrains decoding: the LLM can write a token only if the answer stays a substring of the document.

Typically, an LLM generates the answer token by token, with one forward pass for each token, in a phase called decoding. Naïvely, each forward pass of decoding is punishingly memory bound: the forward pass reads the weights and the KV of the whole document, but computes only one token.1 A popular way to speed up decoding is speculative decoding, proposed at the same time by Leviathan et al.2 and Chen et al.3 In each step, a small, fast draft model proposes the next few tokens, called draft tokens. Then the LLM, called the target model, checks all of them in one forward pass. It accepts draft tokens until it disagrees with one, and then outputs one token of its own.

Speculative decoding can give very large speedups. Intuitively, a memory-bound forward pass spends most of its time reading the weights and the KV, so checking \(k\) draft tokens takes about as long as computing one. If the target model accepts all of them, you get \(k\) tokens for the price of one, a \(k\)-fold speedup. Section 4 gives the more detailed cost model that Quail uses. For more detail on speculative decoding in general though, see some of our favorite resources: Modal’s primer Speculation Is All You Need,4 Fergus Finn’s post on the economics of speculative decoding,5 and the roofline model for speculative decoding in the LLM Engineer’s Almanac, which lets you try different models, GPUs, and batch sizes.6

Contemporary speculative decoding methods like DFlash7 and DSpark8 use another language model with access to the target’s internal representations to draft tokens. In AI.EXTRACT, we can achieve excellent speculative decoding speedups with a much simpler model of language: find the current output in the document and predict that its suffix will follow. Quail finds the answer so far in the document, which we call a match, and copies the next few tokens as the draft.9 Let the draft length \(\gamma\) be the number of tokens that Quail copies. For example, Figure 3 shows the steps of the answer in Figure 2 with \(\gamma = 2\).

Three steps of the answer cat /tests/* with a draft length of 2.
Figure 3. The steps of the answer in Figure 2, with γ = 2 (illustrative). In step 1, the target model picks the first token. In step 2, the target model accepts both draft tokens, then outputs its own token. In step 3, the answer ends, so the target model rejects the draft and ends its turn.

So, how should Quail set \(\gamma\)?

4. Key Idea: Adaptive Draft Lengths for Each Request

Quail chooses a draft length for each request and updates its choice as the query runs. We first model the cost of a draft, then show how Quail learns the model’s inputs during execution. Finally, we compare the approach with prior work. Table 1 lists the symbols.

Table 1. Symbols in the draft-length cost model.
Symbol Meaning
\(\gamma\), \(\gamma^*\) Draft length, and the draft length that minimizes the time per output token
\(E(\gamma)\) Expected number of tokens that a step outputs
\(\alpha\) Acceptance rate, assuming the target model accepts each draft token independently (the classic cost model)
\(\rho\) Time of the KV read of the document in one step, divided by the time to compute one token
\(N\) Number of tokens in the answer
\(S(j)\) Probability that the answer has more than \(j\) tokens left
\(m\) Number of matches of the answer so far in the document
\(\alpha(m)\) Acceptance rate, assuming Quail copies the draft from the first of \(m\) matches: the probability that the first match is correct

4.1 The Cost of a Draft

At each step, the target model checks a draft of \(\gamma\) tokens. A longer draft may produce more output tokens, but takes longer to check. Quail chooses the \(\gamma\) that minimizes expected time per output token:

\[\gamma^* = \arg\min_{\gamma} \frac{\text{time per step}}{E(\gamma)},\]

where \(E(\gamma)\) is the expected number of tokens that a step outputs.

Time per step. Quail copies draft tokens from the document, so producing the draft takes negligible time next to the target model’s forward pass (unlike drafting with an actual neural language model). Since Quail batches tokens from many documents, each forward pass is compute bound; therefore, a step’s time grows with the number of tokens it checks. In units of the time to compute one token, a step takes about \(1 + \gamma + \rho\): the target model’s own token, \(\gamma\) draft tokens, and \(\rho\) for reading the document’s KV cache.

Output tokens per step. The standard mathematical analysis of speculative decoding assumes that each draft token is accepted independently with probability \(\alpha\). Under that assumption, a draft of length \(\gamma\) produces an expected \(E(\gamma) = \sum_{i=0}^{\gamma}\alpha^i\) tokens.10

Extraction workloads severely violate this i.i.d. assumption. Once the target model accepts a draft token, it is at least as likely to accept the next one, because the answer so far now has the same or fewer matches in the document.

So rather than failing at a random token, a draft fails for two main reasons: (1) the answer so far has several matches and Quail copied from the wrong one, or (2) the draft continues past the end of the answer. For (1), let \(m\) be the number of matches for the answer so far, and let \(\alpha(m)\) be the probability that Quail copied from the right match. With one match, \(\alpha(1) = 1\). For (2), let \(S(j)\) be the probability that the answer has more than \(j\) tokens left. The cost model assumes that (1) and (2) are independent.11

As in the standard analysis, \(E(\gamma)\) adds up, for each \(j\), the probability that the step outputs more than \(j\) tokens. For \(j = 0\), the step outputs the target model’s own token whenever the answer continues, with probability \(S(0)\). For \(1 \le j \le \gamma\), the step outputs more than \(j\) tokens only if Quail copied from the right match and the answer has more than \(j\) tokens left, with probability \(\alpha(m)S(j)\). So

\[E(\gamma) = S(0) + \alpha(m) \sum_{j=1}^{\gamma}S(j),\]

and Quail chooses

\[\gamma^* = \arg\min_{\gamma} \frac{1 + \gamma + \rho}{E(\gamma)}.\]

Quail computes \(m\) and estimates \(\rho\) for each document and step. It must learn \(\alpha(m)\) and the answer-length probabilities for each query.

4.2 Learning the Draft Length During Execution

A query processes many documents, so Quail learns from decoding steps and completed requests. It keeps two histograms for each extracted field:

Quail gives both histograms initial counts so it can choose draft lengths before any requests finish. It starts with the equivalent of eight answers averaging 16 tokens. For each \(m\), it starts with eight drafts and an acceptance rate of \(1/m\), as if each match were equally likely to be right. As Quail records more answers and drafts, the initial counts have less influence on the estimates.14

At each step, Quail puts the current estimates into the cost model and chooses \(\gamma\) for that request, given its document, answer so far, and matches. Figure 4 shows the resulting lengths for two queries over legal contracts.

The draft length at each step of CUAD-1 and CUAD-3. Counting only steps that draft, CUAD-1 falls to about 10, and CUAD-3 rises to about 51.
Figure 4. Draft lengths chosen at each step for CUAD-1 and CUAD-3, which process 449 legal contracts. Lengths vary within each query and between the two queries. Among steps that use a draft, the length falls to about 10 in CUAD-1 and rises to about 51 in CUAD-3. Quail chooses γ = 0 for 26% of CUAD-3 steps, likely because many matches make α(m) low. It drafts tokens in 94% of CUAD-1 steps.

Figure 5 shows one reason the choices change: as answers finish, Quail’s estimate of \(P(N > i)\) approaches the curve measured from all answers.

The estimate of P(N > i) during CUAD-1 and CUAD-3, before any answer finishes and after 8, 32, and 128 answers, against the true curve.
Figure 5. Estimated P(N > i) for CUAD-1 and CUAD-3 before any answers finish and after 8, 32, and 128 answers, compared with the final curve.

4.3 Reflecting on Prior Work

Perhaps the best-known recent method that chooses a draft length for each request is DeepSeek’s DSpark,8 whose scheduler is confidence-aware and hardware-aware. DSpark trains a small neural network to estimate how likely each draft token is to be accepted. The scheduler then compares these estimates with the engine’s measured throughput at the current load. DSpark calibrates its estimates offline, before deployment. Interestingly, in Quail we can learn these estimates online (with simple histograms rather than a neural network), separately for each query, as the query runs!

Others have also copied drafts from the input. To fix grammar mistakes in a sentence, where the output is mostly the same as the input, Sun et al.15 copy the input sentence as the draft. N-gram decoding16 copies from the prompt.

SuffixDecoding, by Oliaro et al.,17 both copies drafts and adapts their length. It copies from the prompt and previous outputs, and drafts a tree of continuations when the output has multiple matches. Quail copies the continuation after the first match in the document. SuffixDecoding limits draft length based on match length. Quail chooses the length that minimizes expected time per output token.

When the answer so far has many matches, Quail may draft no tokens and decode one at a time. Instead, Quail could copy the continuation after each match and have the target model check them in one pass. Sequoia, by Chen et al.,18 uses dynamic programming to choose which tokens to draft within a budget, then tunes the draft’s size and depth for the GPU. Quail could incorporate \(\alpha(m)\) into a similar cost model for choosing a draft tree. We leave this to future work.

5. Experiments on QUAIL-B

Setup. All systems run Qwen3-4B (FP8) on one H100 with the same prompts. We test Quail with adaptive draft lengths and fixed lengths \(\gamma \in \{0, 2, 4, 8, 16, 32, 63\}\). Quail limits each extracted answer to 64 tokens. We compare Quail with two vLLM baselines:19

We use two groups of QUAIL-B queries.

Simple queries extract one field per document. They vary document and answer length to show how draft length affects performance (Figure 6).

More realistic queries ask several questions of each document. Figures 7 and 8 use these queries. Figure 6 also includes WRENCH-8.

The best draft length depends on the query. Figure 6 shows the query time with each fixed \(\gamma\), relative to adaptive \(\gamma\). The best fixed \(\gamma\) is 8 for CUAD-1 and IMDB-ACTING, 32 for CUAD-3, and 4 for IMDB-ACTOR and WRENCH-8. A wrong fixed \(\gamma\) costs up to 20%. Adaptive \(\gamma\) needs no tuning for each query: it is within 1% of the best fixed \(\gamma\) on CUAD-1, CUAD-3, and WRENCH-8.

Query time with each fixed draft length relative to adaptive, for contracts and movie reviews with short and long answers.
Figure 6. Query time with each fixed draft length, relative to adaptive, on three QUAIL-B datasets: legal contracts from CUAD, movie reviews from IMDB, and messages of coding agents from Terminal Wrench. The wrong fixed length costs up to 20%.

Quail is faster than vLLM. As Figure 1 shows, Quail is 4.7x faster than naive vLLM and 1.8x faster than tuned vLLM (geometric mean), and costs $0.07 per query on average, compared with $0.69 and $0.13. Figure 7 shows the query time of the three systems on each query. Quail is 2.0–19x faster than naive vLLM, and 1.2–3.0x faster than tuned vLLM. Naive vLLM sends the fields of each document in random order, so its prefix cache rarely still holds the document when the next field arrives. On CUAD-16, where each contract of about 7,500 tokens has 41 fields, naive vLLM computes nearly every prompt from scratch and takes 2,079 s. Sorting the calls removes most of this waste. Against tuned vLLM, on the queries with only extractions, the more fields a query extracts, the larger the gain: 1.2x on WRENCH-8 (2 fields), 1.4x on SEC-9 (4 fields), and 2.0x on CUAD-16 (41 fields). Each field needs its own answer, so a query with more fields spends more of its time on decoding. N-gram speculation removes part of the gap, but the draft length of vLLM is fixed. The gain is largest on SEC-10 (3.0x), because vLLM runs one operator at a time: vLLM computes each filing again for the filters, the classification, and the extractions. Quail computes each filing once for all of them.23

Query time of naive vLLM, tuned vLLM, and Quail on WRENCH-8, SEC-9, SEC-10, and CUAD-16. Quail takes 36.1, 55.0, 52.4, and 108.3 seconds; tuned vLLM is 1.22, 1.37, 2.95, and 1.99 times slower; naive vLLM is 2.02, 3.73, 3.26, and 19.19 times slower.
Figure 7. Query time of naive vLLM, tuned vLLM, and Quail on four queries that ask several questions of each document. Quail is the fastest on every query. The bar of naive vLLM on CUAD-16 is cut off at 300 s.

Quail runs at 38–42% of the speed of light. As in How to Cost Your AI-Powered Filters,24 the speed-of-light (SoL) time of a query is a lower bound from the peak FLOPs and the memory bandwidth of one H100. To extract a field from a document, the SoL time counts two forward passes. The first pass prefills the document and the question, and picks the first token of the answer. The second pass checks the whole answer at once, as if speculation drafted the entire answer perfectly on the first try. We use the most stringent and extreme version of the SoL so that we never fool ourselves into complacency. If there are several fields, we count the prefill of the document only once. Figure 8 compares the throughput of each system, in tokens per second, with the SoL throughput. Quail reaches 38–42% of the SoL throughput, tuned vLLM 14–32%, and naive vLLM 2–20%.

Tokens per second of naive vLLM, tuned vLLM, and Quail, against the speed of light, on WRENCH-8, SEC-9, SEC-10, and CUAD-16. Quail reaches 39%, 41%, 42%, and 38% of the speed of light.
Figure 8. Tokens per second of each system on the queries of Figure 7, against the speed of light. Each system gets the tokens of the speed-of-light run: each document once, and each question and answer. The percentages are the shares of the speed-of-light throughput.

Where does the shortfall from the SoL come from? One source is host overhead,25 which shows up as gaps between kernels on the GPU and is reported as lower GPU kernel utilization.26 In a 5-second window in the middle of IMDB-ACTING, GPU kernel utilization is 93%, which suggests that more careful management of work on the host would cut a few percentage points off the shortfall.

Another important source is rejected draft tokens. To reach the SoL as defined above, Quail would need to predict all output tokens from the first token, perfectly and instantly, on every query. Of course, if we could do that, we would just run that system instead of the target model! We suspect that rejected draft tokens account for a large part of the remaining shortfall, but we have not measured this yet. If that is right, improving drafting methods strikes us as a better target for research than chasing host overhead, which costs only a few percentage points.

6. Conclusion and Future Work

In this article, we walked through how Quail executes AI.EXTRACT, a SQL function that extracts the answers to questions from documents. We use speculative decoding to accelerate AI.EXTRACT, with drafts copied from the document, and a draft length that adapts to the document and to the answer lengths that Quail learns while the query runs. On extraction tasks over agent traces, SEC filings, and legal contracts, Quail is up to 3.0x faster than tuned vLLM, and up to 19x faster than naive vLLM.

Looking forward, it would be great if we didn’t have to decode answers token by token at all. Perhaps one can train a “System One”-like model, in the spirit of TypeSafe’s Jev27 and Cloudflare’s Clairic,28 that points to the start and the end of each answer in the document in one forward pass. If this kind of research is interesting to you, reach out! In the meantime, try out Quail, and if you like what we’re building, give it a star on GitHub!

Acknowledgements

We thank Modal for sponsoring the compute used in this research.

Notes

1. Modal. What does it mean to be memory-bound? GPU Glossary.

2. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Speculative Decoding. In International Conference on Machine Learning (ICML), 2023.

3. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023.

4. Modal. Speculation Is All You Need. Modal Blog, June 2026.

5. Fergus Finn. The Economics of Speculative Decoding. Blog post, June 2026.

6. Modal. Spec Dec Roofline Model. LLM Engineer’s Almanac.

7. Z Lab, Modal, and SGLang Teams. The Next Generation of Speculative Decoding: DFlash and Spec V2. LMSYS Blog, June 2026.

8. Xin Cheng, Xingkai Yu, Chenze Shao, et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026.

9. N-gram decoding also drafts by copying from the input. N-gram decoding finds a match of the last few output tokens in the input, and drafts the input tokens after the match. N-gram decoding works for tasks where the output sometimes copies the input, such as summarization and code editing. In AI.EXTRACT, the answer always copies the document.

10. Here is the intuition. Term \(j\) is the probability that the step outputs more than \(j\) tokens. The step always outputs the own token of the target model, so term 0 is 1. The step outputs more than \(j\) tokens only if the target model accepts the first \(j\) draft tokens, which has probability \(\alpha^j\). Leviathan et al. write the sum in closed form, as \((1 - \alpha^{\gamma+1}) / (1 - \alpha)\).

11. Note that \(\alpha(m)\) is the probability that the whole draft is correct, up to the end of the answer. Assuming that (1) and (2) are independent means that this probability does not depend on how long the answer is.

12. The histogram has 65 bins.

13. Quail puts the values of \(m\) into 11 buckets, from 1 to 1000 on a log scale.

14. The lengths of the 8 initial answers follow a geometric distribution with a mean of 16 tokens. If the answer has \(k\) tokens so far, then \(S(j) = P(N > k + j \mid N \ge k)\).

15. Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021.

16. Apoorv Saxena. Prompt Lookup Decoding. GitHub, November 2023.

17. Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

18. Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding. arXiv:2402.12374, 2024.

19. Both vLLM baselines run vLLM 0.29.0 with up to 25,305 tokens and 4,096 requests per forward pass, 91% of the GPU memory, and prefix caching. Naive vLLM shuffles the calls with a fixed seed. Tuned vLLM also uses speculative_config={"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}, the example in the vLLM documentation. vLLM writes the answer freely, without the constraint that the answer is a substring of the document. Both baselines run filters and classifications one operator at a time, before the extractions.

20. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. In NeurIPS Datasets and Benchmarks, 2021.

21. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of ACL, 2011.

22. Ivan Bercovich, Ivgeni Segal, Kexun Zhang, Shashwat Saxena, Aditi Raghunathan, and Ziqian Zhong. Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories. 2026.

23. The KV regret, i.e., the input tokens that a system computes beyond the minimum, is 5.0%, 5.0%, 0.9%, and 15.8% for Quail on WRENCH-8, SEC-9, SEC-10, and CUAD-16; 3.4%, 2.8%, 63.8%, and 7.9% for tuned vLLM; and 43.3%, 71.3%, 68.6%, and 96.0% for naive vLLM.

24. Arnav Dhariya and Shreya Shankar. How to Cost Your AI-Powered Filters. Full Stack Data Lab Blog, October 2026.

25. Modal. Host Overhead Is Killing Your Inference Efficiency. Modal Blog.

26. Modal. “I Paid for the Whole GPU, I Am Going to Use the Whole GPU”: A High-Level Guide to GPU Utilization. Modal Blog.

27. Diogo Almeida. Introducing System One Models & Jev. TypeSafe AI Blog, September 2026.

28. Alex Reneau, Michelle Chen, and Kevin Flansburg. Introducing Clairic, a New System One Class of Extraction Models. Cloudflare Blog, October 2026.

Cite this article

@misc{shankar2026aiextract,
  title = {Copy That: Faster AI-Powered Extraction by Drafting from the Document},
  author = {Shankar, Shreya and Frye, Charles},
  year = {2026},
  month = nov,
  url = {https://fsdatalab.github.io/blog/ai-extract/}
}