Does the order of my requests matter for vLLM prefix caching in an offline batch job?
Yes. Quail plans KV reuse from the query, so you don't order requests by hand. In our AI.EXTRACT experiments, naive vLLM sent each document's questions in random order and took 2,079 s on CUAD-16, which asks 41 questions of each contract. If you use vLLM directly, sort the requests so a document's questions sit together; in the same experiments, that removed most of the waste.
How it works. Prefix caching keeps the KV of recently seen prompt prefixes in GPU memory, so a later request with the same prefix, such as the same document, can skip recomputing it. Memory is limited, so the cache evicts entries, usually the least recently used. If a document's questions are spread across the batch, its KV is often evicted before the next question arrives and must be computed again. Quail plans KV reuse from the query, so it knows which requests share a document.
Sources: Copy That? Faster AI-Powered Extraction by Drafting from the Document; Building an Ultra-High Throughput AI-SQL Engine
Guide: What Is AI-SQL?
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- How do I reuse the KV cache when I ask many questions about the same long document?
- How do I force an LLM to answer with only one of my class labels?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?