When does speculative decoding actually help, and does it still help at large batch sizes?
Speculative decoding helps when decoding is memory-bound and the drafts are cheap and usually right; at large batches the extra verify tokens cost real compute, so the draft length matters. In our AI.EXTRACT work in Quail, we draft by copying tokens from the document and pick a draft length for each request as the query runs. The best fixed length was 4 on some queries and 32 on others, and a wrong fixed length cost up to 20%.
How it works. An LLM normally writes one token per forward pass, and each pass reads all the model weights and the document's KV from GPU memory, so decoding is limited by memory reads, not arithmetic. Speculative decoding guesses the next few tokens, called the draft, and checks them all in one forward pass; if the model agrees, you get several tokens for the cost of one. Long drafts pay off when they're usually right; when they're often wrong, the extra checked tokens are wasted work. Quail learns, as the query runs, how often drafts are accepted and how long answers are, and picks a draft length for each request.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- How do I reuse the KV cache when I ask many questions about the same long document?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I force an LLM to answer with only one of my class labels?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?