How do I force an LLM to answer with only one of my class labels?
Write the query with Quail's AI.CLASSIFY, which restricts the model to your labels: it builds a trie of the label tokens and runs the model only where labels branch apart. With 10 support-ticket labels, that averages 1.5 decode requests per document, and on 12 classification queries Quail was 1.8x faster per query than vLLM generating and parsing the label. Without Quail, you can use structured or constrained decoding in vLLM or an LLM API to limit the output to your labels.
How it works. A trie, or prefix tree, stores each label as a path of tokens; labels that start with the same words share the start of their path. Quail only asks the model to choose where paths branch apart. For example, with the labels "shipping delay", "shipping damage", and "shipping address change", once the model picks "shipping" and then "address", the word "change" is already determined, so Quail skips that decode step. Constrained decoding in vLLM or an API also limits output to valid labels, but it typically still decodes every token.
Sources: To Classify, Try a Trie Tree
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- How do I reuse the KV cache when I ask many questions about the same long document?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?