How do I get a probability for each label from an LLM classifier, when labels are more than one token?
Quail's AI.CLASSIFY can return a probability for every label. Its trie_tree method scores each label only up to the word where it branches away from the other labels, then normalizes over all labels, so long labels aren't penalized. Quail also has a method that puts a letter before each label, so each label is one token, and it picks the cheaper method for your model and labels.
How it works. A label like "refund request" spans several tokens, and the model gives a probability for each one. Multiplying all of them gives longer labels lower scores even when the model favors them. trie_tree scores each label only up to the point where it splits from the other labels, since later tokens add no information, then divides by the sum over all labels so the probabilities add up to 1.
Sources: To Classify, Try a Trie Tree
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- How do I reuse the KV cache when I ask many questions about the same long document?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I force an LLM to answer with only one of my class labels?
- How can I estimate how long LLM inference over a dataset will take on a given GPU?