How can I estimate how long LLM inference over a dataset will take on a given GPU?

Our post How to Cost Your AI-Powered Filters works through a roofline estimate: count the FLOPs and the bytes moved, then divide by the GPU's peak compute and memory bandwidth. For Qwen3-4B on an H100, one true/false filter over 5,000 IMDB reviews comes out to 6.64 seconds, and a forward pass needs only about 296 tokens to become compute-bound. Quail uses these estimates in its cost model.

How it works. Every step of a forward pass does arithmetic and moves bytes between GPU memory and the compute units. The roofline model estimates a step's time as the slower of the two: arithmetic divided by peak compute, or bytes divided by memory bandwidth. A step is compute-bound when the arithmetic is slower and memory-bound otherwise. Processing more tokens per forward pass reuses the same weights for more work, which is why about 296 tokens per pass is enough to pass the H100's FP8 ridge point.

Sources: How to Cost Your AI-Powered Filters

Guide: What Is AI-SQL?

More questions on this topic

All questions about using AI for data and knowledge work