Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
It works best when the output copies the input, as in extraction, and helps little for freely written text. In Quail's AI.EXTRACT, every answer is a substring of the document, so we copy the draft from the first match in the document and learn, per field, how often drafts are accepted and how long answers run. Quail was 1.8x faster on average than vLLM with n-gram speculation and sorted prompts, and 4.7x faster than naive vLLM.
How it works. Prompt lookup decoding drafts tokens by finding the text written so far in the prompt and guessing that what followed it there comes next. In extraction, every answer is copied from the document, so these guesses are usually right. Quail also constrains decoding so the answer must stay a substring of the document, and it sets the draft length for each request from what it has learned about that field.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- How do I reuse the KV cache when I ask many questions about the same long document?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I force an LLM to answer with only one of my class labels?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?