How do I reuse the KV cache when I ask many questions about the same long document?
Quail handles this for you: it plans KV reuse from the query and keeps a document's KV only while a later operator will read it. On AGENT-5, which runs three classifications over the same traces, 64% of the tokens vLLM computed were recomputations. With a general engine such as vLLM, put the document first and the question last, send a document's questions together, and turn on prefix caching; an LRU cache can still evict the document before the next question arrives.
How it works. When an LLM reads a document, it stores intermediate results for every token, called the KV cache, so it doesn't redo that work when it writes the answer. If you ask a second question about the same document, the engine can reuse that KV instead of reading the document again, but only if it's still in GPU memory. Quail knows from the query which operators will read each document, so it keeps the KV exactly until the last one runs.
Sources: To Classify, Try a Trie Tree; Building an Ultra-High Throughput AI-SQL Engine
Guide: What Is AI-SQL?
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I force an LLM to answer with only one of my class labels?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?