How can I estimate how long LLM inference over a dataset will take on a given GPU?
Our post How to Cost Your AI-Powered Filters works through a roofline estimate: count the FLOPs and the bytes moved, then divide by the GPU's peak compute and memory bandwidth. For Qwen3-4B on an H100, one true/false filter over 5,000 IMDB reviews comes out to 6.64 seconds, and a forward pass needs only about 296 tokens to become compute-bound. Quail uses these estimates in its cost model.
How it works. Every step of a forward pass does arithmetic and moves bytes between GPU memory and the compute units. The roofline model estimates a step's time as the slower of the two: arithmetic divided by peak compute, or bytes divided by memory bandwidth. A step is compute-bound when the arithmetic is slower and memory-bound otherwise. Processing more tokens per forward pass reuses the same weights for more work, which is why about 296 tokens per pass is enough to pass the H100's FP8 ridge point.
Sources: How to Cost Your AI-Powered Filters
Guide: What Is AI-SQL?
More questions on this topic
- What is the fastest, cheapest way to run an LLM over every row of a large dataset?
- My vLLM batch job has thousands of requests but the GPU is still underused. Why?
- When does speculative decoding actually help, and does it still help at large batch sizes?
- Is prompt lookup (n-gram) speculative decoding worth it for extraction or other tasks that copy from the input?
- How do I reuse the KV cache when I ask many questions about the same long document?
- Does the order of my requests matter for vLLM prefix caching in an offline batch job?
- How do I force an LLM to answer with only one of my class labels?
- How do I get a probability for each label from an LLM classifier, when labels are more than one token?