<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Full Stack Data Lab</title>
  <subtitle>A CS research lab led by Shreya Shankar, moving to Carnegie Mellon University in 2027. We build open-source data systems so AI can power data analysis: DocETL, Quail, and DocWriter.</subtitle>
  <link href="https://fsdatalab.github.io/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://fsdatalab.github.io/" rel="alternate" type="text/html"/>
  <id>https://fsdatalab.github.io/</id>
  <updated>2026-11-09T00:00:00+00:00</updated>
  <icon>https://fsdatalab.github.io/assets/logo.png</icon>
  <entry>
    <title>Copy That? Faster AI-Powered Extraction by Drafting from the Document</title>
    <link href="https://fsdatalab.github.io/blog/ai-extract/" rel="alternate" type="text/html"/>
    <id>https://fsdatalab.github.io/blog/ai-extract/</id>
    <published>2026-11-09T00:00:00+00:00</published>
    <updated>2026-11-09T00:00:00+00:00</updated>
    <author><name>Shreya Shankar</name><uri>https://www.sh-reya.com/</uri></author>
    <author><name>Charles Frye</name></author>
    <summary>How Quail speeds up LLM extraction with drafts copied from the document, a draft length chosen for each request, and up to 19x speedups over naive vLLM.</summary>
    <content type="html">&lt;aside class=&quot;tldr&quot;&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Our implementation of &lt;code&gt;AI.EXTRACT&lt;/code&gt; in &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt; is &lt;strong&gt;4.7x faster on average (up to 19x)&lt;/strong&gt; than a naive vLLM baseline, and &lt;strong&gt;1.8x faster on average (up to 3.0x)&lt;/strong&gt; than a hand-written, well-tuned vLLM baseline. The key technique is a novel “inference-time-learned” adaptive draft length, used for speculative decoding with draft tokens copied from the document. Read this post for the details!&lt;/aside&gt;

&lt;figure id=&quot;figure-1&quot; style=&quot;width: min(38.75rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/summary-speedup.svg?v=2&quot; alt=&quot;Speedup over naive vLLM and average cost per query on four QUAIL-B queries. Naive vLLM: 1.0x and $0.69 per query. Tuned vLLM: 2.6x and $0.13. Quail: 4.7x and $0.07.&quot; /&gt;
&lt;figcaption&gt;Figure 1. Speedup over naive vLLM (geometric mean, left) and average cost per query on an H100 at $3.95 per hour (right), on four QUAIL-B queries that ask several questions of each document. Section 5 describes the baselines and the queries.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;nav class=&quot;post-toc&quot; aria-label=&quot;Table of contents&quot;&gt;
&lt;strong&gt;Contents&lt;/strong&gt;
&lt;ol&gt;
  &lt;li&gt;&lt;a href=&quot;#1-introduction&quot;&gt;Introduction.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#2-aiextract-in-quail&quot;&gt;&lt;code&gt;AI.EXTRACT&lt;/code&gt; in Quail.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#3-extracting-with-an-off-the-shelf-llm&quot;&gt;Extracting with an Off-the-Shelf LLM.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#4-key-idea-adaptive-and-per-request-draft-lengths&quot;&gt;Key Idea: Adaptive Draft Lengths for Each Request.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#41-the-cost-of-a-draft&quot;&gt;The Cost of a Draft.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#42-learning-the-length-gamma-during-execution&quot;&gt;Learning the Draft Length During Execution.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#43-reflecting-on-prior-work&quot;&gt;Reflecting on Prior Work.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#5-experiments-on-quail-b&quot;&gt;Experiments on QUAIL-B.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#6-conclusion-and-future-work&quot;&gt;Conclusion and Future Work.&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/nav&gt;

&lt;h2 id=&quot;1-introduction&quot;&gt;1. Introduction&lt;/h2&gt;

&lt;p&gt;Extraction is one of the most common AI operations over data: given a document and a question, return the part of the document that answers it. For example, we might want to, from contracts, extract the governing law; from clinical notes, extract drug doses; from agent traces, extract signs of reward hacking (e.g., commands that peek at the grader code).&lt;/p&gt;

&lt;p&gt;We implemented extraction as the function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; in &lt;a href=&quot;/blog/introducing-quail/&quot;&gt;Quail&lt;/a&gt;, our open-source SQL engine for AI functions. This post explains how Quail runs &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; as fast as possible on NVIDIA GPUs.&lt;/p&gt;

&lt;p&gt;We’ll cover:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;#2-aiextract-in-quail&quot;&gt;What &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; does in Quail&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#3-extracting-with-an-off-the-shelf-llm&quot;&gt;How an off-the-shelf LLM performs extraction, and our trick to speed it up&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#4-key-idea-adaptive-and-per-request-draft-lengths&quot;&gt;How Quail adapts to each query and document as the query runs&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#5-experiments-on-quail-b&quot;&gt;Results on QUAIL-B compared with vLLM&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;2-aiextract-in-quail&quot;&gt;2. AI.EXTRACT in Quail&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; is a SQL function that takes in a document and a set of questions, and returns a struct with the answers, one field for each question. Each answer is a substring of the document. If the document does not answer a question, the field is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NULL&lt;/code&gt;. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; can also return the start and end of each answer (i.e., the span) in the document.&lt;/p&gt;

&lt;p&gt;For example, suppose that a table &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;messages&lt;/code&gt; has one row for each message generated by a coding agent, such as the thoughts of the agent, its commands, and their output. Suppose we want to find messages pertaining to reward hacking, i.e., messages where the agent cheats instead of solving the task, e.g., by peeking at the tests. The following query extracts the plan to hack or cheat and the command that cheats:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
       &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;EXTRACT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;text&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;s1&quot;&gt;&apos;plan&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;Which thought indicates that the agent plans to cheat?&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;s1&quot;&gt;&apos;command&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;Which command peeks at the tests or their answers?&apos;&lt;/span&gt;
       &lt;span class=&quot;p&quot;&gt;})&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cheat&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;messages&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;3-extracting-with-an-off-the-shelf-llm&quot;&gt;3. Extracting with an Off-the-Shelf LLM&lt;/h2&gt;

&lt;p&gt;Suppose that we want to run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt; with an off-the-shelf LLM (i.e., any autoregressive LLM, not a model trained specifically for extraction). The simplest way is to send the LLM one request for each document and field, and get one answer for each request, as in Figure 2.&lt;/p&gt;

&lt;figure id=&quot;figure-2&quot; style=&quot;width: min(48rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/extract-request.svg?v=3&quot; alt=&quot;One AI.EXTRACT request: a message of a coding agent, the question of the command field, the instruction, and the answer cat /tests/*, copied from the message.&quot; /&gt;
&lt;figcaption&gt;Figure 2. One request of the query in Section 2, for the &lt;code&gt;command&lt;/code&gt; field, on one message of a coding agent from Terminal Wrench (shortened). After &lt;code&gt;Phrase:&lt;/code&gt;, the LLM writes the answer &lt;code&gt;cat /tests/*&lt;/code&gt;, copied from the commands in the message.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;The answer must be a substring of the document (which trivially includes the empty string). However, an LLM can write any text. So, Quail constrains decoding: the LLM can write a token only if the answer stays a substring of the document.&lt;/p&gt;

&lt;p&gt;Typically, an LLM generates the answer token by token, with one forward pass for each token, in a phase called &lt;em&gt;decoding&lt;/em&gt;. Naïvely, each forward pass of decoding is punishingly &lt;a href=&quot;https://modal.com/gpu-glossary/perf/memory-bound&quot;&gt;memory bound&lt;/a&gt;: the forward pass reads the weights and the KV of the whole document, but computes only one token.&lt;sup&gt;&lt;a href=&quot;#note-1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; A popular way to speed up decoding is &lt;em&gt;speculative decoding&lt;/em&gt;, proposed at the same time by Leviathan et al.&lt;sup&gt;&lt;a href=&quot;#note-2&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; and Chen et al.&lt;sup&gt;&lt;a href=&quot;#note-3&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; In each step, a small, fast &lt;em&gt;draft model&lt;/em&gt; proposes the next few tokens, called &lt;em&gt;draft tokens&lt;/em&gt;. Then the LLM, called the &lt;em&gt;target model&lt;/em&gt;, checks all of them in one forward pass. It accepts draft tokens until it disagrees with one, and then outputs one token of its own.&lt;/p&gt;

&lt;p&gt;Speculative decoding can give very large speedups. Intuitively, a memory-bound forward pass spends most of its time reading the weights and the KV, so checking \(k\) draft tokens takes about as long as computing one. If the target model accepts all of them, you get \(k\) tokens for the price of one, a \(k\)-fold speedup. Section 4 gives the more detailed cost model that Quail uses. For more detail on speculative decoding in general though, see some of our favorite resources: Modal’s primer &lt;a href=&quot;https://modal.com/blog/spec-is-all-u-need&quot;&gt;Speculation Is All You Need&lt;/a&gt;,&lt;sup&gt;&lt;a href=&quot;#note-4&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; Fergus Finn’s post on &lt;a href=&quot;https://fergusfinn.com/blog/economics-of-speculative-decoding/&quot;&gt;the economics of speculative decoding&lt;/a&gt;,&lt;sup&gt;&lt;a href=&quot;#note-5&quot;&gt;5&lt;/a&gt;&lt;/sup&gt; and the &lt;a href=&quot;https://modal.com/llm-almanac/spec-dec-roofline&quot;&gt;roofline model for speculative decoding&lt;/a&gt; in the LLM Engineer’s Almanac, which lets you try different models, GPUs, and batch sizes.&lt;sup&gt;&lt;a href=&quot;#note-6&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Contemporary speculative decoding methods like DFlash&lt;sup&gt;&lt;a href=&quot;#note-7&quot;&gt;7&lt;/a&gt;&lt;/sup&gt; and DSpark&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt; use another language model with access to the target’s internal representations to draft tokens. In &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt;, we can achieve excellent speculative decoding speedups with a much simpler model of language: &lt;em&gt;find the current output in the document and predict that its suffix will follow.&lt;/em&gt; Quail finds the answer so far in the document, which we call a &lt;em&gt;match&lt;/em&gt;, and copies the next few tokens as the draft.&lt;sup&gt;&lt;a href=&quot;#note-9&quot;&gt;9&lt;/a&gt;&lt;/sup&gt; Let the draft length \(\gamma\) be the number of tokens that Quail copies. For example, Figure 3 shows the steps of the answer in Figure 2 with \(\gamma = 2\).&lt;/p&gt;

&lt;figure id=&quot;figure-3&quot; style=&quot;width: min(45rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/decode-steps.svg?v=2&quot; alt=&quot;Three steps of the answer cat /tests/* with a draft length of 2.&quot; /&gt;
&lt;figcaption&gt;Figure 3. The steps of the answer in Figure 2, with &lt;em&gt;γ&lt;/em&gt; = 2 (illustrative). In step 1, the target model picks the first token. In step 2, the target model accepts both draft tokens, then outputs its own token. In step 3, the answer ends, so the target model rejects the draft and ends its turn.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;So, how should Quail set \(\gamma\)?&lt;/p&gt;

&lt;h2 id=&quot;4-key-idea-adaptive-draft-lengths-for-each-request&quot;&gt;4. Key Idea: Adaptive Draft Lengths for Each Request&lt;/h2&gt;

&lt;p&gt;Quail chooses a draft length for each request and updates its choice as the query runs. We first model the cost of a draft, then show how Quail learns the model’s inputs during execution. Finally, we compare the approach with prior work. Table 1 lists the symbols.&lt;/p&gt;

&lt;details class=&quot;collapsible-section proof-card&quot;&gt;
  &lt;summary id=&quot;table-1&quot;&gt;Table 1. Symbols in the draft-length cost model.&lt;/summary&gt;

  &lt;div class=&quot;table-wrap&quot;&gt;

    &lt;table&gt;
      &lt;thead&gt;
        &lt;tr&gt;
          &lt;th&gt;Symbol&lt;/th&gt;
          &lt;th&gt;Meaning&lt;/th&gt;
        &lt;/tr&gt;
      &lt;/thead&gt;
      &lt;tbody&gt;
        &lt;tr&gt;
          &lt;td&gt;\(\gamma\), \(\gamma^*\)&lt;/td&gt;
          &lt;td&gt;Draft length, and the draft length that minimizes the time per output token&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(E(\gamma)\)&lt;/td&gt;
          &lt;td&gt;Expected number of tokens that a step outputs&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(\alpha\)&lt;/td&gt;
          &lt;td&gt;Acceptance rate, assuming the target model accepts each draft token independently (the classic cost model)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(\rho\)&lt;/td&gt;
          &lt;td&gt;Time of the KV read of the document in one step, divided by the time to compute one token&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(N\)&lt;/td&gt;
          &lt;td&gt;Number of tokens in the answer&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(S(j)\)&lt;/td&gt;
          &lt;td&gt;Probability that the answer has more than \(j\) tokens left&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(m\)&lt;/td&gt;
          &lt;td&gt;Number of matches of the answer so far in the document&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(\alpha(m)\)&lt;/td&gt;
          &lt;td&gt;Acceptance rate, assuming Quail copies the draft from the first of \(m\) matches: the probability that the first match is correct&lt;/td&gt;
        &lt;/tr&gt;
      &lt;/tbody&gt;
    &lt;/table&gt;

  &lt;/div&gt;
&lt;/details&gt;

&lt;h3 id=&quot;41-the-cost-of-a-draft&quot;&gt;4.1 The Cost of a Draft&lt;/h3&gt;

&lt;p&gt;At each step, the target model checks a draft of \(\gamma\) tokens. A longer draft may produce more output tokens, but takes longer to check. Quail chooses the \(\gamma\) that minimizes expected time per output token:&lt;/p&gt;

\[\gamma^* = \arg\min_{\gamma} \frac{\text{time per step}}{E(\gamma)},\]

&lt;p&gt;where \(E(\gamma)\) is the expected number of tokens that a step outputs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Time per step.&lt;/strong&gt; Quail copies draft tokens from the document, so producing the draft takes negligible time next to the target model’s forward pass (unlike drafting with an actual neural language model). Since Quail batches tokens from many documents, each forward pass is compute bound; therefore, a step’s time grows with the number of tokens it checks. In units of the time to compute one token, a step takes about \(1 + \gamma + \rho\): the target model’s own token, \(\gamma\) draft tokens, and \(\rho\) for reading the document’s KV cache.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Output tokens per step.&lt;/strong&gt; The standard mathematical analysis of speculative decoding assumes that each draft token is accepted independently with probability \(\alpha\). Under that assumption, a draft of length \(\gamma\) produces an expected \(E(\gamma) = \sum_{i=0}^{\gamma}\alpha^i\) tokens.&lt;sup&gt;&lt;a href=&quot;#note-10&quot;&gt;10&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Extraction workloads severely violate this i.i.d. assumption. Once the target model accepts a draft token, it is at least as likely to accept the next one, because the answer so far now has the same or fewer matches in the document.&lt;/p&gt;

&lt;p&gt;So rather than failing at a random token, a draft fails for two main reasons: (1) the answer so far has several matches and Quail copied from the wrong one, or (2) the draft continues past the end of the answer. For (1), let \(m\) be the number of matches for the answer so far, and let \(\alpha(m)\) be the probability that Quail copied from the right match. With one match, \(\alpha(1) = 1\). For (2), let \(S(j)\) be the probability that the answer has more than \(j\) tokens left. The cost model assumes that (1) and (2) are independent.&lt;sup&gt;&lt;a href=&quot;#note-11&quot;&gt;11&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;As in the standard analysis, \(E(\gamma)\) adds up, for each \(j\), the probability that the step outputs more than \(j\) tokens. For \(j = 0\), the step outputs the target model’s own token whenever the answer continues, with probability \(S(0)\). For \(1 \le j \le \gamma\), the step outputs more than \(j\) tokens only if Quail copied from the right match and the answer has more than \(j\) tokens left, with probability \(\alpha(m)S(j)\). So&lt;/p&gt;

\[E(\gamma) = S(0) + \alpha(m) \sum_{j=1}^{\gamma}S(j),\]

&lt;p&gt;and Quail chooses&lt;/p&gt;

\[\gamma^* = \arg\min_{\gamma} \frac{1 + \gamma + \rho}{E(\gamma)}.\]

&lt;p&gt;Quail computes \(m\) and estimates \(\rho\) for each document and step. It must learn \(\alpha(m)\) and the answer-length probabilities for each query.&lt;/p&gt;

&lt;h3 id=&quot;42-learning-the-draft-length-during-execution&quot;&gt;4.2 Learning the Draft Length During Execution&lt;/h3&gt;

&lt;p&gt;A query processes many documents, so Quail learns from decoding steps and completed requests. It keeps two histograms for each extracted field:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;em&gt;Answer lengths.&lt;/em&gt; When a request finishes, Quail records its answer length.&lt;sup&gt;&lt;a href=&quot;#note-12&quot;&gt;12&lt;/a&gt;&lt;/sup&gt; The histogram estimates \(P(N &amp;gt; i)\), where \(N\) is the total answer length. Quail uses these estimates and the answer’s current length to estimate how likely the answer is to continue.&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;Match counts.&lt;/em&gt; After a drafted step, Quail records \(m\) and whether the target model accepted the first draft token.&lt;sup&gt;&lt;a href=&quot;#note-13&quot;&gt;13&lt;/a&gt;&lt;/sup&gt; For each value of \(m\), Quail estimates \(\alpha(m)\) as the fraction of drafts whose first token was accepted.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Quail gives both histograms initial counts so it can choose draft lengths before any requests finish. It starts with the equivalent of eight answers averaging 16 tokens. For each \(m\), it starts with eight drafts and an acceptance rate of \(1/m\), as if each match were equally likely to be right. As Quail records more answers and drafts, the initial counts have less influence on the estimates.&lt;sup&gt;&lt;a href=&quot;#note-14&quot;&gt;14&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;At each step, Quail puts the current estimates into the cost model and chooses \(\gamma\) for that request, given its document, answer so far, and matches. Figure 4 shows the resulting lengths for two queries over legal contracts.&lt;/p&gt;

&lt;figure id=&quot;figure-4&quot; style=&quot;width: min(46rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/draft-length-over-query.svg?v=7&quot; alt=&quot;The draft length at each step of CUAD-1 and CUAD-3. Counting only steps that draft, CUAD-1 falls to about 10, and CUAD-3 rises to about 51.&quot; /&gt;
&lt;figcaption&gt;Figure 4. Draft lengths chosen at each step for CUAD-1 and CUAD-3, which process 449 legal contracts. Lengths vary within each query and between the two queries. Among steps that use a draft, the length falls to about 10 in CUAD-1 and rises to about 51 in CUAD-3. Quail chooses &lt;em&gt;γ&lt;/em&gt; = 0 for 26% of CUAD-3 steps, likely because many matches make &lt;em&gt;α&lt;/em&gt;(&lt;em&gt;m&lt;/em&gt;) low. It drafts tokens in 94% of CUAD-1 steps.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Figure 5 shows one reason the choices change: as answers finish, Quail’s estimate of \(P(N &amp;gt; i)\) approaches the curve measured from all answers.&lt;/p&gt;

&lt;figure id=&quot;figure-5&quot; style=&quot;width: min(52rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/answer-length-curve.svg?v=2&quot; alt=&quot;The estimate of P(N &amp;gt; i) during CUAD-1 and CUAD-3, before any answer finishes and after 8, 32, and 128 answers, against the true curve.&quot; /&gt;
&lt;figcaption&gt;Figure 5. Estimated &lt;em&gt;P&lt;/em&gt;(&lt;em&gt;N&lt;/em&gt; &amp;gt; &lt;em&gt;i&lt;/em&gt;) for CUAD-1 and CUAD-3 before any answers finish and after 8, 32, and 128 answers, compared with the final curve.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;43-reflecting-on-prior-work&quot;&gt;4.3 Reflecting on Prior Work&lt;/h3&gt;

&lt;p&gt;Perhaps the best-known recent method that chooses a draft length for each request is DeepSeek’s DSpark,&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt; whose scheduler is confidence-aware and hardware-aware. DSpark trains a small neural network to estimate how likely each draft token is to be accepted. The scheduler then compares these estimates with the engine’s measured throughput at the current load. DSpark calibrates its estimates offline, before deployment. Interestingly, in Quail we can learn these estimates online (with simple histograms rather than a neural network), &lt;em&gt;separately&lt;/em&gt; for each query, as the query runs!&lt;/p&gt;

&lt;p&gt;Others have also copied drafts from the input. To fix grammar mistakes in a sentence, where the output is mostly the same as the input, Sun et al.&lt;sup&gt;&lt;a href=&quot;#note-15&quot;&gt;15&lt;/a&gt;&lt;/sup&gt; copy the input sentence as the draft. N-gram decoding&lt;sup&gt;&lt;a href=&quot;#note-16&quot;&gt;16&lt;/a&gt;&lt;/sup&gt; copies from the prompt.&lt;/p&gt;

&lt;p&gt;SuffixDecoding, by Oliaro et al.,&lt;sup&gt;&lt;a href=&quot;#note-17&quot;&gt;17&lt;/a&gt;&lt;/sup&gt; both copies drafts and adapts their length. It copies from the prompt and previous outputs, and drafts a tree of continuations when the output has multiple matches. Quail copies the continuation after the first match in the document. SuffixDecoding limits draft length based on match length. Quail chooses the length that minimizes expected time per output token.&lt;/p&gt;

&lt;p&gt;When the answer so far has many matches, Quail may draft no tokens and decode one at a time. Instead, Quail could copy the continuation after each match and have the target model check them in one pass. Sequoia, by Chen et al.,&lt;sup&gt;&lt;a href=&quot;#note-18&quot;&gt;18&lt;/a&gt;&lt;/sup&gt; uses dynamic programming to choose which tokens to draft within a budget, then tunes the draft’s size and depth for the GPU. Quail could incorporate \(\alpha(m)\) into a similar cost model for choosing a draft tree. We leave this to future work.&lt;/p&gt;

&lt;h2 id=&quot;5-experiments-on-quail-b&quot;&gt;5. Experiments on QUAIL-B&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Setup.&lt;/strong&gt; All systems run Qwen3-4B (FP8) on one H100 with the same prompts. We test Quail with adaptive draft lengths and fixed lengths \(\gamma \in \{0, 2, 4, 8, 16, 32, 63\}\). Quail limits each extracted answer to 64 tokens. We compare Quail with two vLLM baselines:&lt;sup&gt;&lt;a href=&quot;#note-19&quot;&gt;19&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Naive vLLM&lt;/strong&gt;, our naive baseline. We compile each query into LLM calls, one for every field of every document. Then we pass all of the calls to stock vLLM’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LLM.generate&lt;/code&gt; in one batch, in random order. vLLM generates each answer one token at a time.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Tuned vLLM&lt;/strong&gt;, our hand-tuned baseline. Same as naive vLLM, but we sort the calls so that the fields of one document are next to each other, so that vLLM’s prefix caching computes each document once. We also turn on n-gram speculation: vLLM finds the last few answer tokens in the prompt and drafts the next five tokens.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We use two groups of QUAIL-B queries.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Simple queries&lt;/em&gt; extract one field per document. They vary document and answer length to show how draft length affects performance (Figure 6).&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;CUAD-1 and CUAD-3 read 449 legal contracts from CUAD,&lt;sup&gt;&lt;a href=&quot;#note-20&quot;&gt;20&lt;/a&gt;&lt;/sup&gt; about 7,500 tokens each. They extract the governing law (short answers) and the liability cap clause (long answers), respectively.&lt;/li&gt;
  &lt;li&gt;IMDB-ACTOR and IMDB-ACTING read 5,000 movie reviews from IMDB,&lt;sup&gt;&lt;a href=&quot;#note-21&quot;&gt;21&lt;/a&gt;&lt;/sup&gt; about 300 tokens each. They ask which actor the reviewer names (short answers) and what the reviewer says about the acting (long answers), respectively.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;More realistic queries&lt;/em&gt; ask several questions of each document. Figures 7 and 8 use these queries. Figure 6 also includes WRENCH-8.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;WRENCH-8 is the query from Section 2. It reads 3,454 coding agent messages from Terminal Wrench,&lt;sup&gt;&lt;a href=&quot;#note-22&quot;&gt;22&lt;/a&gt;&lt;/sup&gt; about 1,200 tokens each. Each message contains one turn of the agent’s thoughts, commands, and command output, as in Figure 2.&lt;/li&gt;
  &lt;li&gt;SEC-9 and SEC-10 read 1,056 &lt;a href=&quot;https://huggingface.co/datasets/TeraflopAI/SEC-EDGAR&quot;&gt;SEC EDGAR 8-K filings&lt;/a&gt; from 2023 or later, each 2,400 to 22,000 tokens long. Companies file 8-Ks to report important events. SEC-9 extracts who is involved, the amount, the sentence describing the event, and what changes. SEC-10 uses two filters to keep material agreements with a dollar amount, classifies the agreement, and extracts the other party and the amount.&lt;/li&gt;
  &lt;li&gt;CUAD-16 extracts all 41 CUAD clause types from the contracts used in CUAD-1.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The best draft length depends on the query.&lt;/strong&gt; Figure 6 shows the query time with each fixed \(\gamma\), relative to adaptive \(\gamma\). The best fixed \(\gamma\) is 8 for CUAD-1 and IMDB-ACTING, 32 for CUAD-3, and 4 for IMDB-ACTOR and WRENCH-8. A wrong fixed \(\gamma\) costs up to 20%. Adaptive \(\gamma\) needs no tuning for each query: it is within 1% of the best fixed \(\gamma\) on CUAD-1, CUAD-3, and WRENCH-8.&lt;/p&gt;

&lt;figure id=&quot;figure-6&quot; style=&quot;width: min(60rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/fixed-vs-adaptive.svg?v=15&quot; alt=&quot;Query time with each fixed draft length relative to adaptive, for contracts and movie reviews with short and long answers.&quot; /&gt;
&lt;figcaption&gt;Figure 6. Query time with each fixed draft length, relative to adaptive, on three QUAIL-B datasets: legal contracts from CUAD, movie reviews from IMDB, and messages of coding agents from Terminal Wrench. The wrong fixed length costs up to 20%.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Quail is faster than vLLM.&lt;/strong&gt; As Figure 1 shows, Quail is 4.7x faster than naive vLLM and 1.8x faster than tuned vLLM (geometric mean), and costs $0.07 per query on average, compared with $0.69 and $0.13. Figure 7 shows the query time of the three systems on each query. Quail is 2.0–19x faster than naive vLLM, and 1.2–3.0x faster than tuned vLLM. Naive vLLM sends the fields of each document in random order, so its prefix cache rarely still holds the document when the next field arrives. On CUAD-16, where each contract of about 7,500 tokens has 41 fields, naive vLLM computes nearly every prompt from scratch and takes 2,079 s. Sorting the calls removes most of this waste. Against tuned vLLM, on the queries with only extractions, the more fields a query extracts, the larger the gain: 1.2x on WRENCH-8 (2 fields), 1.4x on SEC-9 (4 fields), and 2.0x on CUAD-16 (41 fields). Each field needs its own answer, so a query with more fields spends more of its time on decoding. N-gram speculation removes part of the gap, but the draft length of vLLM is fixed. The gain is largest on SEC-10 (3.0x), because vLLM runs one operator at a time: vLLM computes each filing again for the filters, the classification, and the extractions. Quail computes each filing once for all of them.&lt;sup&gt;&lt;a href=&quot;#note-23&quot;&gt;23&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;figure id=&quot;figure-7&quot; style=&quot;width: min(52rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/quail-vs-vllm.svg?v=15&quot; alt=&quot;Query time of naive vLLM, tuned vLLM, and Quail on WRENCH-8, SEC-9, SEC-10, and CUAD-16. Quail takes 36.1, 55.0, 52.4, and 108.3 seconds; tuned vLLM is 1.22, 1.37, 2.95, and 1.99 times slower; naive vLLM is 2.02, 3.73, 3.26, and 19.19 times slower.&quot; /&gt;
&lt;figcaption&gt;Figure 7. Query time of naive vLLM, tuned vLLM, and Quail on four queries that ask several questions of each document. Quail is the fastest on every query. The bar of naive vLLM on CUAD-16 is cut off at 300 s.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Quail runs at 38–42% of the speed of light.&lt;/strong&gt; As in &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;How to Cost Your AI-Powered Filters&lt;/a&gt;,&lt;sup&gt;&lt;a href=&quot;#note-24&quot;&gt;24&lt;/a&gt;&lt;/sup&gt; the &lt;em&gt;speed-of-light&lt;/em&gt; (SoL) time of a query is a lower bound from the peak FLOPs and the memory bandwidth of one H100. To extract a field from a document, the SoL time counts two forward passes. The first pass prefills the document and the question, and picks the first token of the answer. The second pass checks the whole answer at once, as if speculation drafted the entire answer perfectly on the first try. We use the most stringent and extreme version of the SoL so that we never fool ourselves into complacency. If there are several fields, we count the prefill of the document only once. Figure 8 compares the throughput of each system, in tokens per second, with the SoL throughput. Quail reaches 38–42% of the SoL throughput, tuned vLLM 14–32%, and naive vLLM 2–20%.&lt;/p&gt;

&lt;figure id=&quot;figure-8&quot; style=&quot;width: min(52rem, calc(100vw - 3rem));&quot;&gt;
&lt;img src=&quot;/assets/blog/ai-extract/throughput-vs-sol.svg?v=14&quot; alt=&quot;Tokens per second of naive vLLM, tuned vLLM, and Quail, against the speed of light, on WRENCH-8, SEC-9, SEC-10, and CUAD-16. Quail reaches 39%, 41%, 42%, and 38% of the speed of light.&quot; /&gt;
&lt;figcaption&gt;Figure 8. Tokens per second of each system on the queries of Figure 7, against the speed of light. Each system gets the tokens of the speed-of-light run: each document once, and each question and answer. The percentages are the shares of the speed-of-light throughput.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Where does the shortfall from the SoL come from? One source is &lt;a href=&quot;https://modal.com/blog/host-overhead-inference-efficiency&quot;&gt;host overhead&lt;/a&gt;,&lt;sup&gt;&lt;a href=&quot;#note-25&quot;&gt;25&lt;/a&gt;&lt;/sup&gt; which shows up as gaps between kernels on the GPU and is reported as lower &lt;a href=&quot;https://modal.com/blog/gpu-utilization-guide&quot;&gt;GPU kernel utilization&lt;/a&gt;.&lt;sup&gt;&lt;a href=&quot;#note-26&quot;&gt;26&lt;/a&gt;&lt;/sup&gt; In a 5-second window in the middle of IMDB-ACTING, GPU kernel utilization is 93%, which suggests that more careful management of work on the host would cut a few percentage points off the shortfall.&lt;/p&gt;

&lt;p&gt;Another important source is rejected draft tokens. To reach the SoL as defined above, Quail would need to predict all output tokens from the first token, perfectly and instantly, on every query. Of course, if we could do that, we would just run that system instead of the target model! We suspect that rejected draft tokens account for a large part of the remaining shortfall, but we have not measured this yet. If that is right, improving drafting methods strikes us as a better target for research than chasing host overhead, which costs only a few percentage points.&lt;/p&gt;

&lt;h2 id=&quot;6-conclusion-and-future-work&quot;&gt;6. Conclusion and Future Work&lt;/h2&gt;

&lt;p&gt;In this article, we walked through how Quail executes &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt;, a SQL function that extracts the answers to questions from documents. We use speculative decoding to accelerate &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt;, with drafts copied from the document, and a draft length that adapts to the document and to the answer lengths that Quail learns while the query runs. On extraction tasks over agent traces, SEC filings, and legal contracts, Quail is up to 3.0x faster than tuned vLLM, and up to 19x faster than naive vLLM.&lt;/p&gt;

&lt;p&gt;Looking forward, it would be great if we didn’t have to decode answers token by token at all. Perhaps one can train a “System One”-like model, in the spirit of TypeSafe’s &lt;a href=&quot;https://typesafe.ai/blog/introducing-system-one-models-and-jev&quot;&gt;Jev&lt;/a&gt;&lt;sup&gt;&lt;a href=&quot;#note-27&quot;&gt;27&lt;/a&gt;&lt;/sup&gt; and Cloudflare’s &lt;a href=&quot;https://blog.cloudflare.com/clairic-extraction-model&quot;&gt;Clairic&lt;/a&gt;,&lt;sup&gt;&lt;a href=&quot;#note-28&quot;&gt;28&lt;/a&gt;&lt;/sup&gt; that points to the start and the end of each answer in the document in one forward pass. If this kind of research is interesting to you, reach out! In the meantime, try out &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt;, and if you like what we’re building, give it a star on GitHub!&lt;/p&gt;

&lt;h2 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h2&gt;

&lt;p&gt;We thank &lt;a href=&quot;https://modal.com/&quot;&gt;Modal&lt;/a&gt; for sponsoring the compute used in
this research.&lt;/p&gt;

&lt;h2 id=&quot;notes&quot;&gt;Notes&lt;/h2&gt;

&lt;p&gt;&lt;span id=&quot;note-1&quot;&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/span&gt; Modal. &lt;a href=&quot;https://modal.com/gpu-glossary/perf/memory-bound&quot;&gt;What does it mean to be memory-bound?&lt;/a&gt; &lt;em&gt;GPU Glossary&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-2&quot;&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/span&gt; Yaniv Leviathan, Matan Kalman, and Yossi Matias. &lt;a href=&quot;https://arxiv.org/abs/2211.17192&quot;&gt;Fast Inference from Transformers via Speculative Decoding&lt;/a&gt;. In &lt;em&gt;International Conference on Machine Learning (ICML)&lt;/em&gt;, 2023.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-3&quot;&gt;&lt;strong&gt;3.&lt;/strong&gt;&lt;/span&gt; Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. &lt;a href=&quot;https://arxiv.org/abs/2302.01318&quot;&gt;Accelerating Large Language Model Decoding with Speculative Sampling&lt;/a&gt;. arXiv:2302.01318, 2023.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-4&quot;&gt;&lt;strong&gt;4.&lt;/strong&gt;&lt;/span&gt; Modal. &lt;a href=&quot;https://modal.com/blog/spec-is-all-u-need&quot;&gt;Speculation Is All You Need&lt;/a&gt;. &lt;em&gt;Modal Blog&lt;/em&gt;, June 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-5&quot;&gt;&lt;strong&gt;5.&lt;/strong&gt;&lt;/span&gt; Fergus Finn. &lt;a href=&quot;https://fergusfinn.com/blog/economics-of-speculative-decoding/&quot;&gt;The Economics of Speculative Decoding&lt;/a&gt;. Blog post, June 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-6&quot;&gt;&lt;strong&gt;6.&lt;/strong&gt;&lt;/span&gt; Modal. &lt;a href=&quot;https://modal.com/llm-almanac/spec-dec-roofline&quot;&gt;Spec Dec Roofline Model&lt;/a&gt;. &lt;em&gt;LLM Engineer’s Almanac&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-7&quot;&gt;&lt;strong&gt;7.&lt;/strong&gt;&lt;/span&gt; Z Lab, Modal, and SGLang Teams. &lt;a href=&quot;https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/&quot;&gt;The Next Generation of Speculative Decoding: DFlash and Spec V2&lt;/a&gt;. &lt;em&gt;LMSYS Blog&lt;/em&gt;, June 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-8&quot;&gt;&lt;strong&gt;8.&lt;/strong&gt;&lt;/span&gt; Xin Cheng, Xingkai Yu, Chenze Shao, et al. &lt;a href=&quot;https://arxiv.org/abs/2607.05147&quot;&gt;DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation&lt;/a&gt;. arXiv:2607.05147, 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-9&quot;&gt;&lt;strong&gt;9.&lt;/strong&gt;&lt;/span&gt; &lt;a href=&quot;https://github.com/apoorvumang/prompt-lookup-decoding&quot;&gt;N-gram decoding&lt;/a&gt; also drafts by copying from the input. N-gram decoding finds a match of the last few output tokens in the input, and drafts the input tokens after the match. N-gram decoding works for tasks where the output sometimes copies the input, such as summarization and code editing. In &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt;, the answer always copies the document.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-10&quot;&gt;&lt;strong&gt;10.&lt;/strong&gt;&lt;/span&gt; Here is the intuition. Term \(j\) is the probability that the step outputs more than \(j\) tokens. The step always outputs the own token of the target model, so term 0 is 1. The step outputs more than \(j\) tokens only if the target model accepts the first \(j\) draft tokens, which has probability \(\alpha^j\). Leviathan et al. write the sum in closed form, as \((1 - \alpha^{\gamma+1}) / (1 - \alpha)\).&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-11&quot;&gt;&lt;strong&gt;11.&lt;/strong&gt;&lt;/span&gt; Note that \(\alpha(m)\) is the probability that the whole draft is correct, up to the end of the answer. Assuming that (1) and (2) are independent means that this probability does not depend on how long the answer is.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-12&quot;&gt;&lt;strong&gt;12.&lt;/strong&gt;&lt;/span&gt; The histogram has 65 bins.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-13&quot;&gt;&lt;strong&gt;13.&lt;/strong&gt;&lt;/span&gt; Quail puts the values of \(m\) into 11 buckets, from 1 to 1000 on a log scale.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-14&quot;&gt;&lt;strong&gt;14.&lt;/strong&gt;&lt;/span&gt; The lengths of the 8 initial answers follow a geometric distribution with a mean of 16 tokens. If the answer has \(k\) tokens so far, then \(S(j) = P(N &amp;gt; k + j \mid N \ge k)\).&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-15&quot;&gt;&lt;strong&gt;15.&lt;/strong&gt;&lt;/span&gt; Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. &lt;a href=&quot;https://aclanthology.org/2021.acl-long.462&quot;&gt;Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding&lt;/a&gt;. In &lt;em&gt;Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)&lt;/em&gt;, 2021.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-16&quot;&gt;&lt;strong&gt;16.&lt;/strong&gt;&lt;/span&gt; Apoorv Saxena. &lt;a href=&quot;https://github.com/apoorvumang/prompt-lookup-decoding&quot;&gt;Prompt Lookup Decoding&lt;/a&gt;. GitHub, November 2023.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-17&quot;&gt;&lt;strong&gt;17.&lt;/strong&gt;&lt;/span&gt; Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. &lt;a href=&quot;https://arxiv.org/abs/2411.04975&quot;&gt;SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications&lt;/a&gt;. In &lt;em&gt;Advances in Neural Information Processing Systems (NeurIPS)&lt;/em&gt;, 2025.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-18&quot;&gt;&lt;strong&gt;18.&lt;/strong&gt;&lt;/span&gt; Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. &lt;a href=&quot;https://arxiv.org/abs/2402.12374&quot;&gt;Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding&lt;/a&gt;. arXiv:2402.12374, 2024.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-19&quot;&gt;&lt;strong&gt;19.&lt;/strong&gt;&lt;/span&gt; Both vLLM baselines run vLLM 0.29.0 with up to 25,305 tokens and 4,096 requests per forward pass, 91% of the GPU memory, and prefix caching. Naive vLLM shuffles the calls with a fixed seed. Tuned vLLM also uses &lt;code&gt;speculative_config={&quot;method&quot;: &quot;ngram&quot;, &quot;num_speculative_tokens&quot;: 5, &quot;prompt_lookup_max&quot;: 4}&lt;/code&gt;, the example in the vLLM documentation. vLLM writes the answer freely, without the constraint that the answer is a substring of the document. Both baselines run filters and classifications one operator at a time, before the extractions.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-20&quot;&gt;&lt;strong&gt;20.&lt;/strong&gt;&lt;/span&gt; Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. &lt;a href=&quot;https://arxiv.org/abs/2103.06268&quot;&gt;CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review&lt;/a&gt;. In &lt;em&gt;NeurIPS Datasets and Benchmarks&lt;/em&gt;, 2021.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-21&quot;&gt;&lt;strong&gt;21.&lt;/strong&gt;&lt;/span&gt; Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. &lt;a href=&quot;https://aclanthology.org/P11-1015&quot;&gt;Learning Word Vectors for Sentiment Analysis&lt;/a&gt;. In &lt;em&gt;Proceedings of ACL&lt;/em&gt;, 2011.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-22&quot;&gt;&lt;strong&gt;22.&lt;/strong&gt;&lt;/span&gt; Ivan Bercovich, Ivgeni Segal, Kexun Zhang, Shashwat Saxena, Aditi Raghunathan, and Ziqian Zhong. &lt;a href=&quot;https://github.com/few-sh/terminal-wrench&quot;&gt;Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories&lt;/a&gt;. 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-23&quot;&gt;&lt;strong&gt;23.&lt;/strong&gt;&lt;/span&gt; The &lt;em&gt;KV regret&lt;/em&gt;, i.e., the input tokens that a system computes beyond the minimum, is 5.0%, 5.0%, 0.9%, and 15.8% for Quail on WRENCH-8, SEC-9, SEC-10, and CUAD-16; 3.4%, 2.8%, 63.8%, and 7.9% for tuned vLLM; and 43.3%, 71.3%, 68.6%, and 96.0% for naive vLLM.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-24&quot;&gt;&lt;strong&gt;24.&lt;/strong&gt;&lt;/span&gt; Arnav Dhariya and Shreya Shankar. &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;How to Cost Your AI-Powered Filters&lt;/a&gt;. &lt;em&gt;Full Stack Data Lab Blog&lt;/em&gt;, October 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-25&quot;&gt;&lt;strong&gt;25.&lt;/strong&gt;&lt;/span&gt; Modal. &lt;a href=&quot;https://modal.com/blog/host-overhead-inference-efficiency&quot;&gt;Host Overhead Is Killing Your Inference Efficiency&lt;/a&gt;. &lt;em&gt;Modal Blog&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-26&quot;&gt;&lt;strong&gt;26.&lt;/strong&gt;&lt;/span&gt; Modal. &lt;a href=&quot;https://modal.com/blog/gpu-utilization-guide&quot;&gt;“I Paid for the Whole GPU, I Am Going to Use the Whole GPU”: A High-Level Guide to GPU Utilization&lt;/a&gt;. &lt;em&gt;Modal Blog&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-27&quot;&gt;&lt;strong&gt;27.&lt;/strong&gt;&lt;/span&gt; Diogo Almeida. &lt;a href=&quot;https://typesafe.ai/blog/introducing-system-one-models-and-jev&quot;&gt;Introducing System One Models &amp;amp; Jev&lt;/a&gt;. &lt;em&gt;TypeSafe AI Blog&lt;/em&gt;, September 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-28&quot;&gt;&lt;strong&gt;28.&lt;/strong&gt;&lt;/span&gt; Alex Reneau, Michelle Chen, and Kevin Flansburg. &lt;a href=&quot;https://blog.cloudflare.com/clairic-extraction-model&quot;&gt;Introducing Clairic, a New System One Class of Extraction Models&lt;/a&gt;. &lt;em&gt;Cloudflare Blog&lt;/em&gt;, October 2026.&lt;/p&gt;

&lt;h2 id=&quot;cite-this-article&quot;&gt;Cite this article&lt;/h2&gt;

&lt;div class=&quot;bibtex-block&quot;&gt;
  &lt;p&gt;&lt;button class=&quot;copy-bibtex&quot; type=&quot;button&quot;&gt;Copy BibTeX&lt;/button&gt;&lt;/p&gt;

  &lt;div class=&quot;language-bibtex highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nc&quot;&gt;@misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;shankar2026aiextract&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;title&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Copy That: Faster AI-Powered Extraction by Drafting from the Document}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;author&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Shankar, Shreya and Frye, Charles}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;year&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{2026}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;month&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;nov&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;url&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{https://fsdatalab.github.io/blog/ai-extract/}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

&lt;/div&gt;
</content>
  </entry>
  <entry>
    <title>Five Principles for Generating Readable AI Text in Bulk</title>
    <link href="https://fsdatalab.github.io/blog/agent-text-at-scale/" rel="alternate" type="text/html"/>
    <id>https://fsdatalab.github.io/blog/agent-text-at-scale/</id>
    <published>2026-10-29T00:00:00+00:00</published>
    <updated>2026-10-29T00:00:00+00:00</updated>
    <author><name>Shreya Shankar</name><uri>https://www.sh-reya.com/</uri></author>
    <summary>Five principles for making large amounts of AI-generated text, e.g., in dashboards or websites, easy for humans to read, with real examples and fixes.</summary>
    <content type="html">&lt;aside class=&quot;tldr&quot;&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Feeds of AI-generated text, like agent session titles or dashboard summaries, are hard to read in bulk. We give five principles, with real examples and fixes, for making them easy to scan.&lt;/aside&gt;

&lt;nav class=&quot;post-toc&quot; aria-label=&quot;Table of contents&quot;&gt;
&lt;strong&gt;Contents&lt;/strong&gt;
&lt;ol&gt;
  &lt;li&gt;&lt;a href=&quot;#the-problem&quot;&gt;The Problem.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#checklist&quot;&gt;Checklist.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#1-match-how-human-readers-parse-sentences&quot;&gt;Match How Human Readers Parse Sentences.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#2-put-the-most-important-words-first&quot;&gt;Put the Most Important Words First.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#3-use-the-same-structure-and-the-same-vocabulary&quot;&gt;Use the Same Structure and the Same Vocabulary.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#4-use-variation-only-to-show-a-meaningful-difference&quot;&gt;Use Variation Only to Show a Meaningful Difference.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#5-align-the-connotation-of-verbs-with-the-purpose-of-the-dashboard&quot;&gt;Align the Connotation of Verbs with the Purpose of the Dashboard.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#takeaways&quot;&gt;Takeaways.&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/nav&gt;

&lt;h2 id=&quot;the-problem&quot;&gt;The Problem&lt;/h2&gt;

&lt;p&gt;Thanks to modern AI, we humans find ourselves reading AI-generated text at an unprecedented scale! Many of our daily driver products show feeds of such text, e.g., in Claude Code or Codex, there is a sidebar with AI-generated titles of our past agent sessions; on an iPhone, Apple Intelligence now groups and summarizes “stacks” of notifications. People also generate many temporary dashboards and web pages with AI, and share them broadly.&lt;/p&gt;

&lt;p&gt;Unfortunately, such feeds of AI-generated texts are quite difficult for humans to read. This isn’t very surprising, as it’s well-known that text items need to be &lt;em&gt;predictable&lt;/em&gt; and &lt;em&gt;consistent&lt;/em&gt; for humans to read many of them quickly.&lt;/p&gt;

&lt;p&gt;In this blog post, informed by our recent experience working with AI to generate text dashboards (and consume lots of AI-generated text in dashboards), we present a checklist of 5 principles for effective feeds of AI-generated text.&lt;/p&gt;

&lt;h2 id=&quot;checklist&quot;&gt;Checklist&lt;/h2&gt;

&lt;h3 id=&quot;1-match-how-human-readers-parse-sentences&quot;&gt;1. Match How Human Readers Parse Sentences&lt;/h3&gt;

&lt;p&gt;Interestingly, humans parse text quite predictably and simply, and we have known this for quite some time. In 1970, Thomas Bever, a psycholinguist who studies how people understand sentences, found that English readers use the “NVN” strategy: they take the first noun as the subject, the next verb as its action, and the next noun as its object.&lt;sup&gt;&lt;a href=&quot;#note-1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; For example, in “The agent fixed the login bug,” readers take “agent” as the subject, “fixed” as the action, and “bug” as the object.&lt;/p&gt;

&lt;p&gt;As a consequence, if text does not follow the NVN pattern, humans need to do multiple passes over the text, which is quite annoying.&lt;sup&gt;&lt;a href=&quot;#note-2&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; For example, in “Login bug fixed by the agent,” readers first take “bug” as the subject and “fixed” as its action, as if the bug fixed something. Only at “by the agent” do readers find the real subject, and then they parse the text again.&lt;/p&gt;

&lt;p&gt;To render a list of text items with as little cognitive effort for readers as possible, here is a rough procedure to follow:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Start each item with its subject or its verb. If the subject is the same in all items, drop it and start with the verb.&lt;/li&gt;
  &lt;li&gt;Put the first verb of each item in the tense that best matches the reader’s expectations: past, continuous, or future.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; Many of us use a coding agent app every day, e.g., Claude or Codex. The sidebar of the app shows a list of agent sessions. &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt; (left) shows a real example from the Claude sidebar. The titles mix nouns (“KV persistence”), gerunds (“Switching to main”), and imperatives (“Fine-tune model on personal writing”). On the right are the same titles, rewritten with the procedure above.&lt;/p&gt;

&lt;p&gt;All sessions have the same subject, the agent, so by the procedure above, each title should start with a verb. Then, because each session is active, each verb is in the continuous form, e.g., “Measuring vLLM performance.”&lt;/p&gt;

&lt;figure id=&quot;figure-1&quot; style=&quot;width: min(52rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/agent-text-at-scale/session-titles.svg&quot; alt=&quot;Seven session titles as written in a chat sidebar, next to the same titles rewritten to start with a verb.&quot; /&gt;
  &lt;figcaption&gt;Figure 1. Seven session titles from the author&apos;s Claude sidebar, stylized (left), and the same titles rewritten to start with a verb in the continuous form (right).&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;2-put-the-most-important-words-first&quot;&gt;2. Put the Most Important Words First&lt;/h3&gt;

&lt;p&gt;Jakob Nielsen, a very famous usability researcher who studies how people read websites, used eye tracking to find that readers scan web pages in an “F” shape: they read the start of each line and skip most of the rest&lt;sup&gt;&lt;a href=&quot;#note-3&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; — and, readers of a list often see only the first two words of each item.&lt;sup&gt;&lt;a href=&quot;#note-4&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; As such, we should:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Find the words that make each item different from the other items.&lt;/li&gt;
  &lt;li&gt;Move those words as close to the start of the item as possible.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; As shown in &lt;a href=&quot;#figure-2&quot;&gt;Figure 2&lt;/a&gt;, a bad way to write order email subjects is to put the status last, e.g., “Your order #1001 has shipped.” Then every subject starts with the same words. But Amazon puts the status first, e.g., &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Ordered: &quot;Bounty Essentials...&quot;&lt;/code&gt;.&lt;/p&gt;

&lt;figure id=&quot;figure-2&quot; style=&quot;width: min(59.5rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/agent-text-at-scale/order-emails.svg&quot; alt=&quot;Four order emails in an inbox with the status last, next to four order emails with the status first.&quot; /&gt;
  &lt;figcaption&gt;Figure 2. Order emails in an inbox, stylized. When the status is last (left), the first words are the same in every email. When the status is first, as in Amazon order emails (right), the first word tells the emails apart.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;3-use-the-same-structure-and-the-same-vocabulary&quot;&gt;3. Use the Same Structure and the Same Vocabulary&lt;/h3&gt;

&lt;p&gt;In another interesting psycholinguistic study from the 1980s, Lyn Frazier and her colleagues measured how fast people read sentences with two clauses joined by “and.” Readers read the second clause faster when it had the same structure as the first clause, e.g., when both clauses were active or both were passive.&lt;sup&gt;&lt;a href=&quot;#note-5&quot;&gt;5&lt;/a&gt;&lt;/sup&gt; One can directly apply this idea to a list of texts: each item is faster to read when it has the same structure as the items before it.&lt;sup&gt;&lt;a href=&quot;#note-6&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Moreover, it is well-known that a piece of text should not use two different words for the same concept.&lt;sup&gt;&lt;a href=&quot;#note-7&quot;&gt;7&lt;/a&gt;&lt;/sup&gt; So, in a list of texts:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Use one structure in every item.&lt;/li&gt;
  &lt;li&gt;Use one word for each concept.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; We present one example that does &lt;em&gt;not&lt;/em&gt; follow this principle, followed by one that does. For our bad example, we show the AI autofill feature in Notion — which writes one summary for each row of a table in Notion. We made a table of 10 public GitHub issues from Ollama and let Notion AI fill in a Summary column.&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt; As shown in &lt;a href=&quot;#figure-3&quot;&gt;Figure 3&lt;/a&gt;, the summaries are actually hard to quickly skim. The summaries inconsistently start with “The issue reports that,” “The page reports,” or an action (“Pressing Ctrl+C”), and they call the same thing an “issue” in some rows and a “page” in another.&lt;/p&gt;

&lt;figure id=&quot;figure-3&quot; style=&quot;width: min(59.75rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/agent-text-at-scale/notion-summaries.svg&quot; alt=&quot;A Notion table of four GitHub issues with summaries that Notion AI wrote. The summaries start in different ways.&quot; /&gt;
  &lt;figcaption&gt;Figure 3. Four rows of a Notion table, stylized. Notion AI wrote each summary from the text of the issue. The highlights mark how each summary starts.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;In contrast, Perplexity’s Discover feed nicely follows this checklist item #3 (as well as item #1).&lt;sup&gt;&lt;a href=&quot;#note-9&quot;&gt;9&lt;/a&gt;&lt;/sup&gt; As shown in &lt;a href=&quot;#figure-4&quot;&gt;Figure 4&lt;/a&gt;, every headline has the same structure: the subject, a verb in the present tense, and then the object, e.g., “Bank of Canada hikes rate to 2.5%” and “Brewers beat Rays in Game 5 for first World Series title.”&lt;/p&gt;

&lt;figure id=&quot;figure-4&quot; style=&quot;width: min(40rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/agent-text-at-scale/discover-headlines.svg&quot; alt=&quot;Four stories from the Perplexity Discover feed, each with a headline that has a subject, a verb in the present tense, and an object.&quot; /&gt;
  &lt;figcaption&gt;Figure 4. Four stories from the Perplexity Discover feed on October 28, 2026, stylized. Every headline has the same structure.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;4-use-variation-only-to-show-a-meaningful-difference&quot;&gt;4. Use Variation Only to Show a Meaningful Difference&lt;/h3&gt;

&lt;p&gt;In 1986, Jock Mackinlay, a visualization researcher (and pioneer of modern data visualization), wrote a paper on how to design charts automatically.&lt;sup&gt;&lt;a href=&quot;#note-10&quot;&gt;10&lt;/a&gt;&lt;/sup&gt; One of its main ideas is &lt;em&gt;expressiveness&lt;/em&gt;: a chart should express all of the facts in the data, and only the facts in the data. E.g., if one bar in a bar chart is red, readers will assume that the color means something — so make sure any variation in bars represents some meaningful variable.&lt;sup&gt;&lt;a href=&quot;#note-11&quot;&gt;11&lt;/a&gt;&lt;/sup&gt; Now, although this research is about charts of structured (i.e., tabular) data, it is not so far-fetched to apply it to unstructured text: if one item in a list differs from the others, e.g., in its length, its structure, or its words, readers will assume that the difference means something.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; LLMs are often used to summarize the “key themes” in user feedback. We ran Claude’s data analysis plugin on 100 recent Duolingo App Store reviews, to see what a frontier model would do.&lt;sup&gt;&lt;a href=&quot;#note-12&quot;&gt;12&lt;/a&gt;&lt;/sup&gt; On the left of &lt;a href=&quot;#figure-5&quot;&gt;Figure 5&lt;/a&gt;, each card adds a different kind of detail. Only the bugs card names an app version (7.146.0), and only the energy card says that some reviews are in Russian and Turkish. So readers cannot tell whether these details are significant: is 7.146.0 the only version with bugs, and do the other themes come only from English reviews?&lt;/p&gt;

&lt;p&gt;On the right, we rewrote every card so that each one has the same components: a sentence about what users say, followed by one example from the reviews. We also show the number of reviews, an important variable, for each theme in a bar chart. Of course, the rewritten cards drop some detail, but readers can still open the reviews of any theme they care about.&lt;/p&gt;

&lt;figure id=&quot;figure-5&quot; class=&quot;figure-themes&quot;&gt;
&lt;div class=&quot;td&quot; data-theme-dashboard=&quot;&quot;&gt;
&lt;div class=&quot;td-col&quot;&gt;&lt;div class=&quot;td-head&quot;&gt;Claude&apos;s version&lt;/div&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-orig&quot; data-i=&quot;0&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Fun, easy, and effective learning&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;74 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Most reviews praise Duolingo as fun, easy, and helpful for learning, often in short 5-star notes. Users cite new vocabulary, pronunciation help, self-paced lessons, and being able to talk with family or friends. Spanish is the most named language, followed by Japanese, Italian, Korean, German, and others. Several of these reviewers still raise a complaint, such as energy limits or cost.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-orig&quot; data-i=&quot;1&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Chess, music, and math courses&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;12 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Chess is a popular new draw, and some users mention music, piano, and math too. Feedback is mostly positive, but there are specific complaints: a checkmate rule that seems wrong, no way for skilled players to skip ahead, a distracting new owl voice and hard-to-see pieces in the chess puzzle, and many clicks between games.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-orig&quot; data-i=&quot;2&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Bugs and technical problems&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;9 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Reports include voice answers not registering, the app freezing (one user lost a streak), the app crashing on launch, finished lessons not counting, leaderboard members disappearing, a new event missing for one family-plan member, and a shared iPad starting a new learner at the wrong level. Most come from the current version, 7.146.0.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-orig&quot; data-i=&quot;3&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Energy and battery limits on lessons&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;8 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;The energy (battery) system, and the older hearts system, limits how much users can practice. Users say they run out fast, must wait or watch ads, and see it as a push to buy Super. Some reviews are in Russian and Turkish, so the complaint shows up across markets. One user suggests earning energy by doing chess, music, or math.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-orig&quot; data-i=&quot;4&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;New language requests&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;3 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users ask for new courses. Two ask for Icelandic, noting that Duolingo already offers other Nordic languages, and one asks for Elvish from The Lord of the Rings.&lt;/span&gt;&lt;/button&gt;
&lt;/div&gt;&lt;div class=&quot;td-col&quot;&gt;&lt;div class=&quot;td-head&quot;&gt;Our version&lt;/div&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-new&quot; data-i=&quot;0&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Fun, easy, and effective learning&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;74 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users love that the app is fun and helps them learn, e.g., new vocabulary and pronunciation.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-new&quot; data-i=&quot;1&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Chess, music, and math courses&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;12 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users enjoy the new courses but complain about problems, e.g., a checkmate rule that seems wrong.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-new&quot; data-i=&quot;2&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Bugs and technical problems&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;9 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users complain about bugs that cost them progress, e.g., a freeze that lost a daily streak.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-new&quot; data-i=&quot;3&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;Energy and battery limits on lessons&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;8 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users complain that they run out of energy fast and must wait to keep practicing.&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-card td-new&quot; data-i=&quot;4&quot;&gt;&lt;span class=&quot;td-row&quot;&gt;&lt;span class=&quot;td-th&quot;&gt;New language requests&lt;/span&gt;&lt;span class=&quot;td-n&quot;&gt;3 reviews&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-sum&quot;&gt;Users request new courses, e.g., Icelandic.&lt;/span&gt;&lt;/button&gt;
&lt;div class=&quot;td-sub&quot;&gt;Reviews per theme&lt;/div&gt;&lt;div class=&quot;td-bars&quot;&gt;
&lt;button type=&quot;button&quot; class=&quot;td-bar&quot; data-i=&quot;0&quot;&gt;&lt;span class=&quot;td-bl&quot;&gt;Fun, easy, and effective learning&lt;/span&gt;&lt;span class=&quot;td-track&quot;&gt;&lt;span class=&quot;td-fill&quot; style=&quot;width:100.0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-bn&quot;&gt;74&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-bar&quot; data-i=&quot;1&quot;&gt;&lt;span class=&quot;td-bl&quot;&gt;Chess, music, and math courses&lt;/span&gt;&lt;span class=&quot;td-track&quot;&gt;&lt;span class=&quot;td-fill&quot; style=&quot;width:16.2%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-bn&quot;&gt;12&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-bar&quot; data-i=&quot;2&quot;&gt;&lt;span class=&quot;td-bl&quot;&gt;Bugs and technical problems&lt;/span&gt;&lt;span class=&quot;td-track&quot;&gt;&lt;span class=&quot;td-fill&quot; style=&quot;width:12.2%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-bn&quot;&gt;9&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-bar&quot; data-i=&quot;3&quot;&gt;&lt;span class=&quot;td-bl&quot;&gt;Energy and battery limits on lessons&lt;/span&gt;&lt;span class=&quot;td-track&quot;&gt;&lt;span class=&quot;td-fill&quot; style=&quot;width:10.8%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-bn&quot;&gt;8&lt;/span&gt;&lt;/button&gt;
&lt;button type=&quot;button&quot; class=&quot;td-bar&quot; data-i=&quot;4&quot;&gt;&lt;span class=&quot;td-bl&quot;&gt;New language requests&lt;/span&gt;&lt;span class=&quot;td-track&quot;&gt;&lt;span class=&quot;td-fill&quot; style=&quot;width:4.1%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;td-bn&quot;&gt;3&lt;/span&gt;&lt;/button&gt;
&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;
&lt;figcaption&gt;Figure 5. A dashboard of key themes in 100 Duolingo App Store reviews, generated by Claude via its data analysis plugin, stylized (left; 5 of 11 themes shown). On the right, the same themes rewritten with one template, with the number of reviews per theme as a bar chart. Click a card or a bar to highlight the same theme in all three views.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;script src=&quot;/assets/js/theme-dashboard.js&quot; defer=&quot;&quot;&gt;&lt;/script&gt;

&lt;h3 id=&quot;5-align-the-connotation-of-verbs-with-the-purpose-of-the-dashboard&quot;&gt;5. Align the Connotation of Verbs with the Purpose of the Dashboard&lt;/h3&gt;

&lt;p&gt;Many dashboards exist to surface one kind of information, e.g., what customers dislike, or what customers like and what is going well. The verbs in a dashboard should make its purpose obvious. For example, a dashboard of customer complaints should use clearly negative verbs, like “complain,” and a dashboard of praise should use clearly positive verbs, like “love.” Readers skim dashboards with very little attention, so a verb with a strong connotation helps readers grasp each item quickly.&lt;sup&gt;&lt;a href=&quot;#note-13&quot;&gt;13&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; In &lt;a href=&quot;#figure-5&quot;&gt;Figure 5&lt;/a&gt;, Claude’s version starts the chess card with “Chess is a popular new draw,” even though most of the card lists complaints. So readers cannot tell at a glance whether the theme is good or bad news. In our version, each card starts with a verb that has an obvious sentiment, e.g., “Users love” for praise and “Users complain” for problems, so readers can tell from the verb alone whether a theme is praise or a complaint.&lt;/p&gt;

&lt;h2 id=&quot;takeaways&quot;&gt;Takeaways&lt;/h2&gt;

&lt;p&gt;In this post, we discussed five principles for making lots of AI-generated text easy to read. While we only discussed simple dashboard-like examples, we believe these principles apply to &lt;em&gt;any&lt;/em&gt; artifact or product that AI generates, e.g., websites. So please, feed this checklist into your coding agents, or make a skill out of it, so your AI-generated texts are easy for humans to read. Page by page, let’s move towards a world free of slop.&lt;/p&gt;

&lt;h2 id=&quot;notes&quot;&gt;Notes&lt;/h2&gt;

&lt;p&gt;&lt;span id=&quot;note-1&quot;&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/span&gt; Thomas G. Bever. The Cognitive Basis for Linguistic Structures. In J. R. Hayes, editor, &lt;em&gt;Cognition and the Development of Language&lt;/em&gt;, pages 279–362. Wiley, New York, 1970. Bever describes the strategy as reading a noun, a verb, and a noun as the actor, the action, and the object of the action.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-2&quot;&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/span&gt; Lyn Frazier and Keith Rayner. &lt;a href=&quot;https://doi.org/10.1016/0010-0285(82)90008-1&quot;&gt;Making and Correcting Errors During Sentence Comprehension: Eye Movements in the Analysis of Structurally Ambiguous Sentences&lt;/a&gt;. &lt;em&gt;Cognitive Psychology&lt;/em&gt;, 14(2):178–210, 1982. In an eye-tracking study, readers of sentences with a misleading start moved their eyes back to earlier words to parse the sentence again.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-3&quot;&gt;&lt;strong&gt;3.&lt;/strong&gt;&lt;/span&gt; Jakob Nielsen. &lt;a href=&quot;https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content-discovered/&quot;&gt;F-Shaped Pattern for Reading Web Content (Original Study)&lt;/a&gt;. Nielsen Norman Group, April 16, 2006. The study tracked the eyes of 232 users as they read thousands of web pages.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-4&quot;&gt;&lt;strong&gt;4.&lt;/strong&gt;&lt;/span&gt; Jakob Nielsen. &lt;a href=&quot;https://www.nngroup.com/articles/first-2-words-a-signal-for-scanning/&quot;&gt;First 2 Words: A Signal for the Scanning Eye&lt;/a&gt;. Nielsen Norman Group, April 6, 2009.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-5&quot;&gt;&lt;strong&gt;5.&lt;/strong&gt;&lt;/span&gt; Lyn Frazier, Lori Taft, Tom Roeper, Charles Clifton, and Kate Ehrlich. &lt;a href=&quot;https://link.springer.com/article/10.3758/BF03198303&quot;&gt;Parallel Structure: A Source of Facilitation in Sentence Comprehension&lt;/a&gt;. &lt;em&gt;Memory &amp;amp; Cognition&lt;/em&gt;, 12(5):421–430, 1984. In 2010, Sturt, Keller, and Dubey found that readers also read a phrase faster when it has the same structure as an earlier phrase, even when “and” does not join the two phrases: Patrick Sturt, Frank Keller, and Amit Dubey. &lt;a href=&quot;https://doi.org/10.1016/j.jml.2010.01.001&quot;&gt;Syntactic Priming in Comprehension: Parallelism Effects with and without Coordination&lt;/a&gt;. &lt;em&gt;Journal of Memory and Language&lt;/em&gt;, 62(4):333–351, 2010.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-6&quot;&gt;&lt;strong&gt;6.&lt;/strong&gt;&lt;/span&gt; Funnily enough, modern data compression algorithms work in a similar way, by exploiting repeated patterns. E.g., gzip stores a pattern the first time it appears and replaces each later copy with a short pointer back to it. So a list with one pattern is easy to compress, both for gzip and for a reader.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-7&quot;&gt;&lt;strong&gt;7.&lt;/strong&gt;&lt;/span&gt; E.g., a core rule of &lt;a href=&quot;https://www.asd-ste100.org/about.html&quot;&gt;ASD Simplified Technical English&lt;/a&gt;, a writing standard for aircraft maintenance manuals, is “one word, one meaning.” Also, Jakob Nielsen’s fourth usability heuristic says that users “should not have to wonder whether different words, situations, or actions mean the same thing.” See Jakob Nielsen. &lt;a href=&quot;https://www.nngroup.com/articles/ten-usability-heuristics/&quot;&gt;10 Usability Heuristics for User Interface Design&lt;/a&gt;. Nielsen Norman Group, 1994.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-8&quot;&gt;&lt;strong&gt;8.&lt;/strong&gt;&lt;/span&gt; We copied 10 open issues from &lt;a href=&quot;https://github.com/ollama/ollama/issues&quot;&gt;ollama/ollama&lt;/a&gt; on October 28, 2026, and added a Notion AI autofill property named “Summary” with its default settings. Figure 4 shows the summaries of issues &lt;a href=&quot;https://github.com/ollama/ollama/issues/19359&quot;&gt;#19359&lt;/a&gt;, &lt;a href=&quot;https://github.com/ollama/ollama/issues/19342&quot;&gt;#19342&lt;/a&gt;, &lt;a href=&quot;https://github.com/ollama/ollama/issues/19375&quot;&gt;#19375&lt;/a&gt;, and &lt;a href=&quot;https://github.com/ollama/ollama/issues/19340&quot;&gt;#19340&lt;/a&gt;, without changes.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-9&quot;&gt;&lt;strong&gt;9.&lt;/strong&gt;&lt;/span&gt; &lt;a href=&quot;https://www.perplexity.ai/discover&quot;&gt;Perplexity Discover&lt;/a&gt;, accessed October 28, 2026.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-10&quot;&gt;&lt;strong&gt;10.&lt;/strong&gt;&lt;/span&gt; Jock Mackinlay. &lt;a href=&quot;https://doi.org/10.1145/22949.22950&quot;&gt;Automating the Design of Graphical Presentations of Relational Information&lt;/a&gt;. &lt;em&gt;ACM Transactions on Graphics&lt;/em&gt;, 5(2):110–141, 1986.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-11&quot;&gt;&lt;strong&gt;11.&lt;/strong&gt;&lt;/span&gt; Jeff Heer. &lt;a href=&quot;https://courses.cs.washington.edu/courses/cse512/24sp/&quot;&gt;CSE 512: Data Visualization&lt;/a&gt;. University of Washington, Spring 2024. See the lectures on data and image models and on visualization design.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-12&quot;&gt;&lt;strong&gt;12.&lt;/strong&gt;&lt;/span&gt; We downloaded the 100 most recent US App Store reviews of Duolingo from Apple’s public review feed on October 28, 2026; they are from October 26 and 27, 2026. Claude ran the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;data:analyze&lt;/code&gt; skill of Claude’s data analysis plugin on them and wrote 11 theme cards. Figure 5 shows five of them, without changes. The word counts in the text are ours; the plugin did not report them.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-13&quot;&gt;&lt;strong&gt;13.&lt;/strong&gt;&lt;/span&gt; It turns out, after asking Claude, that there is related prior work, which we have not read closely. E.g., Kousta, Vinson, and Vigliocco found that people recognize emotion words, positive or negative, faster than neutral words. See Stavroula-Thaleia Kousta, David P. Vinson, and Gabriella Vigliocco. &lt;a href=&quot;https://doi.org/10.1016/j.cognition.2009.06.007&quot;&gt;Emotion Words, Regardless of Polarity, Have a Processing Advantage over Neutral Words&lt;/a&gt;. &lt;em&gt;Cognition&lt;/em&gt;, 112(3):473–481, 2009.&lt;/p&gt;

</content>
  </entry>
  <entry>
    <title>To Classify, Try a Trie Tree</title>
    <link href="https://fsdatalab.github.io/blog/ai-classification/" rel="alternate" type="text/html"/>
    <id>https://fsdatalab.github.io/blog/ai-classification/</id>
    <published>2026-10-12T00:00:00+00:00</published>
    <updated>2026-10-12T00:00:00+00:00</updated>
    <author><name>Shreya Shankar</name><uri>https://www.sh-reya.com/</uri></author>
    <author><name>Charles Frye</name></author>
    <summary>How should an AI-SQL engine classify documents with LLMs at scale? We walk through the ways Quail can do it, and how it picks the cheapest one for a given LLM.</summary>
    <content type="html">&lt;aside class=&quot;tldr&quot;&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; How should an AI-SQL engine classify documents with LLMs at scale? Most systems let the LLM generate an answer and then parse it into a label. But since &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt; runs the LLM inside its own inference engine, we can do &lt;em&gt;much&lt;/em&gt; better &amp;mdash; we can restrict the LLM to the labels and minimize the number of tokens it decodes! On 12 new classification queries in &lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt;, Quail is &lt;strong&gt;1.8x faster&lt;/strong&gt; than a vLLM baseline.&lt;/aside&gt;

&lt;nav class=&quot;post-toc&quot; aria-label=&quot;Table of contents&quot;&gt;
&lt;strong&gt;Contents&lt;/strong&gt;
&lt;ol&gt;
  &lt;li&gt;&lt;a href=&quot;#1-introduction&quot;&gt;Introduction.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#2-aiclassify-in-quail&quot;&gt;&lt;code&gt;AI.CLASSIFY&lt;/code&gt; in Quail.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#3-how-quail-executes-aiclassify&quot;&gt;How Quail Executes &lt;code&gt;AI.CLASSIFY&lt;/code&gt;.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#31-trie_decode-greedily-generate-the-label&quot;&gt;&lt;code&gt;trie_decode&lt;/code&gt;: Greedily Generate the Label.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#32-trie_tree-score-all-labels-at-once&quot;&gt;&lt;code&gt;trie_tree&lt;/code&gt;: Score All Labels at Once.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#33-letters-score-one-token-per-label&quot;&gt;&lt;code&gt;letters&lt;/code&gt;: Score One Token per Label.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#34-keeping-the-document-kv-for-later-operators&quot;&gt;Keeping the Document KV for Later Operators.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#4-pricing-and-picking-a-method&quot;&gt;Pricing and Picking a Method.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#41-workload-and-notation&quot;&gt;Workload and Notation.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#42-cost-of-trie_tree-and-letters&quot;&gt;Cost of &lt;code&gt;trie_tree&lt;/code&gt; and &lt;code&gt;letters&lt;/code&gt;.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#43-cost-of-trie_decode&quot;&gt;Cost of &lt;code&gt;trie_decode&lt;/code&gt;.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#44-picking-a-method&quot;&gt;Picking a Method.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#45-experiments-on-quail-b&quot;&gt;Experiments on QUAIL-B.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#5-aside-decision-models&quot;&gt;Aside: Decision Models.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#6-conclusion&quot;&gt;Conclusion.&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/nav&gt;

&lt;h2 id=&quot;1-introduction&quot;&gt;1. Introduction&lt;/h2&gt;

&lt;p&gt;We recently released &lt;a href=&quot;/blog/introducing-quail/&quot;&gt;Quail&lt;/a&gt;, an execution engine for AI-SQL. One of the most common AI-SQL operations is classification: given a document and a list of labels, pick the label that fits best. E.g., a support team might want to sort every incoming message into “billing”, “refund request”, “shipping delay”, and so on.&lt;/p&gt;

&lt;p&gt;Most systems classify a document the same way they answer any other question. They put the document and the labels into a prompt, let the LLM generate a response, and then parse the response into a label. The documentation of &lt;a href=&quot;https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-classify&quot;&gt;BigQuery’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;https://docs.snowflake.com/en/sql-reference/functions/ai_classify&quot;&gt;Snowflake’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI_CLASSIFY&lt;/code&gt;&lt;/a&gt; describes a similar interface: each function adds a prompt to the input, sends it to an LLM, and returns the category that best fits. But letting the LLM generate freely is both wasteful and fragile — the LLM can spend many tokens on its response, and the response may not even contain a valid label!&lt;/p&gt;

&lt;p&gt;Since Quail runs the LLM inside its own inference engine, it doesn’t have to let the LLM generate freely. Instead, Quail can restrict the LLM to the labels, and it turns out there are several ways to do so, each with a different cost. In this article, we discuss: &lt;strong&gt;For a given LLM, how should Quail execute &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt;?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We’ll go through:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;How &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; works in Quail (&lt;a href=&quot;#2-aiclassify-in-quail&quot;&gt;Section 2&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;Three ways that Quail can execute &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt;: generating the label greedily, scoring all labels at once, and scoring one letter per label (&lt;a href=&quot;#3-how-quail-executes-aiclassify&quot;&gt;Section 3&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;How to estimate the cost of each way, building on &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;our blog post on AI-powered filters&lt;/a&gt;, and how Quail picks the cheapest one (&lt;a href=&quot;#4-pricing-and-picking-a-method&quot;&gt;Section 4&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;An aside on decision models, which are LLMs fine-tuned to pick among answers (&lt;a href=&quot;#5-aside-decision-models&quot;&gt;Section 5&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;2-aiclassify-in-quail&quot;&gt;2. AI.CLASSIFY in Quail&lt;/h2&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; takes a document and a list of labels. It returns one of the labels.&lt;/p&gt;

&lt;p&gt;Here is the query that we use throughout the post, with Qwen3-4B-fp8 as the LLM. It sorts support messages into 10 topics:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
       &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CLASSIFY&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
         &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;body&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;n&quot;&gt;ARRAY&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;billing&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;refund request&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;refund status&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
               &lt;span class=&quot;s1&quot;&gt;&apos;shipping delay&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;shipping damage&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
               &lt;span class=&quot;s1&quot;&gt;&apos;shipping address change&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;product defect&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
               &lt;span class=&quot;s1&quot;&gt;&apos;two-factor authentication issue&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;praise&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;other&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
       &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;topic&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;messages&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The function has three arguments:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;input&lt;/strong&gt;: the document to classify. Here, the input is the text of a support message.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;categories&lt;/strong&gt;: 2 to 255 labels. Each label can have a short description of up to 25 words.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;options&lt;/strong&gt; (not in the example query): an object with optional settings.
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;task_description&lt;/code&gt;: extra instructions for the LLM, up to 50 words.&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;probabilities&lt;/code&gt;: when &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;true&lt;/code&gt;, Quail also returns the column &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;topic_probabilities&lt;/code&gt;, with one probability for each label. The probabilities sum to 1.&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;selectivity&lt;/code&gt;: an estimate of the fraction of rows that a condition on the label keeps, e.g., &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;WHERE topic = &apos;billing&apos;&lt;/code&gt;. The optimizer uses the estimate to plan the query. The estimate does not change the labels.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Quail always returns one of the labels.&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; can also appear in a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;WHERE&lt;/code&gt; clause. E.g., the following query keeps only the messages about refunds:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;messages&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;WHERE&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CLASSIFY&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;body&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ARRAY&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[...])&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;IN&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;refund request&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;refund status&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;A condition on the label uses &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;=&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IN&lt;/code&gt;. When the same classification appears in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SELECT&lt;/code&gt; and in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;WHERE&lt;/code&gt;, Quail runs the classification only once.&lt;/p&gt;

&lt;h2 id=&quot;3-how-quail-executes-aiclassify&quot;&gt;3. How Quail Executes AI.CLASSIFY&lt;/h2&gt;

&lt;p&gt;In database terms, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; is a logical operator. Quail supports three physical implementations of the operator, and the optimizer selects one implementation for each query.&lt;/p&gt;

&lt;p&gt;All three implementations send the LLM one prompt for each document. The prompt starts with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;DOCUMENT:&lt;/code&gt; and the document. Then it has a question with the list of labels. After the prompt comes the token &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, where the LLM scores the first token of the answer. The prompts of the three implementations differ slightly in the question and the label list, as shown in &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The implementations also select the label in different ways. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; generates the label token by token, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; scores every label in one request, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; scores one letter for each label.&lt;/p&gt;

&lt;figure id=&quot;figure-1&quot; style=&quot;width: min(48.5rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/letters-prompt.svg&quot; alt=&quot;The prompt of letters next to the prompt of trie_tree and trie_decode.&quot; /&gt;
  &lt;figcaption&gt;Figure 1. The prompt for &lt;code&gt;trie_decode&lt;/code&gt; and &lt;code&gt;trie_tree&lt;/code&gt; (left) and for &lt;code&gt;letters&lt;/code&gt; (right). &lt;code&gt;letters&lt;/code&gt; puts a letter before each label, and the LLM scores the letters at &lt;code&gt;ANSWER:&lt;/code&gt;.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;31-trie_decode-greedily-generate-the-label&quot;&gt;3.1 trie_decode: Greedily Generate the Label&lt;/h3&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; is the implementation that is closest to what prior systems do. It generates the label one token at a time, as an LLM generates text.&lt;sup&gt;&lt;a href=&quot;#note-1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; At each decode step, the LLM picks the most likely token that continues a label.&lt;/p&gt;

&lt;p&gt;At a high level, we can assemble the set of labels into a &lt;em&gt;trie&lt;/em&gt; (&lt;a href=&quot;#figure-2&quot;&gt;Figure 2&lt;/a&gt;). A trie, also called a prefix tree, stores each label as a path of tokens from the root. Labels that start with the same tokens share the start of their paths, e.g., both refund labels pass through &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt;. The root of the trie is the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt; token. For the rest of the article, we write labels in quotes and tokens in code, e.g., the label “refund request” has the tokens &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;request&lt;/code&gt;.&lt;/p&gt;

&lt;figure id=&quot;figure-2&quot; style=&quot;width: min(50.4rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/label-trie.svg&quot; alt=&quot;The label trie for the 10 support labels.&quot; /&gt;
  &lt;figcaption&gt;Figure 2. The trie of the 10 labels in the running example. The &lt;code&gt;ANSWER:&lt;/code&gt; token is the root. The labels branch at three nodes: &lt;code&gt;ANSWER:&lt;/code&gt;, &lt;code&gt;refund&lt;/code&gt;, and &lt;code&gt;shipping&lt;/code&gt;. Each label ends at a node with a dark outline. E.g., &lt;code&gt;refund&lt;/code&gt; is not a label, but &lt;code&gt;refund&lt;/code&gt; followed by &lt;code&gt;request&lt;/code&gt; is.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;The simple way to generate a label is one token per decode step. At each decode step, the LLM reads the tokens so far and picks the most likely child of the current node in the trie. E.g., to generate “shipping address change”, the LLM could take 3 decode steps. It would pick &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt; at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;address&lt;/code&gt;, and then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;change&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Now, you might realize — if the answer really was “shipping address change”, by the time we are at the second decode step (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;address&lt;/code&gt;), we actually need not decode the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;change&lt;/code&gt; token, because there is no other valid token!&lt;/p&gt;

&lt;p&gt;More generally, the LLM only needs to make a choice where the labels branch apart in the trie, i.e., at a node with more than one child. In our running example, there are only 3 such nodes: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt; (which first word?), &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; (request or status?), and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt; (delay, damage, or address?). Everywhere else, the next token is already determined. So Quail only runs the LLM at the 3 branching nodes, and it stops as soon as there’s just one label left. For instance, if the LLM picks &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;product&lt;/code&gt; at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, the answer must be “product defect”, so one step and we’re done!&lt;/p&gt;

&lt;p&gt;What does this look like when we classify many documents at once? &lt;a href=&quot;#figure-3&quot;&gt;Figure 3&lt;/a&gt; shows the decode steps for a batch of 10 documents. Quail computes each step for all documents in the batch together, in one forward pass. Step 0 happens in the same request as the prompt: the LLM reads the document and picks the first token at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;. Then, each later decode step sends one more request, but only for the documents whose labels aren’t decided yet. In our running example, only the 5 documents that picked &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt; need a step 1, and the other 5 documents are already done.&lt;/p&gt;

&lt;figure id=&quot;figure-3&quot; style=&quot;width: min(69.9rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/decode-rounds.svg&quot; alt=&quot;The decode steps of trie_decode for 10 documents.&quot; /&gt;
  &lt;figcaption&gt;Figure 3. The steps of &lt;code&gt;trie_decode&lt;/code&gt; for a batch of 10 documents, with one forward pass per decode step. Each document gets a different label. Step 0 is part of the same request as the prompt, and it selects the first label token at the &lt;code&gt;ANSWER:&lt;/code&gt; token. Only the 5 documents that selected &lt;code&gt;refund&lt;/code&gt; or &lt;code&gt;shipping&lt;/code&gt; need step 1. A green box is a token that the next step sends back to the LLM. A dashed box is the token that decides the label.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;So &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; is pretty cheap: if every label is equally likely, a document in our running example needs just 1.5 requests on average. But there is a catch — since it is greedy, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; only looks at the tokens on its own path. It never scores the other labels, so it can’t return label probabilities. So, if the user asks for probabilities, Quail must use one of the other methods.&lt;sup&gt;&lt;a href=&quot;#note-2&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h3 id=&quot;32-trie_tree-score-all-labels-at-once&quot;&gt;3.2 trie_tree: Score All Labels at Once&lt;/h3&gt;

&lt;p&gt;What if the user wants probabilities? A simple approach would score every label with its full probability under the LLM. The probability of a label is the product of the probabilities of its tokens. If label \(m\) has the tokens \(x_1, \dots, x_t\), then&lt;/p&gt;

\[P(m) = \prod_{k=1}^{t} P(x_k \mid \text{prompt}, x_1, \dots, x_{k-1}).\]

&lt;p&gt;E.g., for “refund request”,&lt;/p&gt;

\[P(\text{refund request}) = P(\texttt{refund} \mid \text{prompt}) \times P(\texttt{request} \mid \text{prompt}, \texttt{refund}).\]

&lt;p&gt;However, the simple approach would unfairly penalize longer labels. Each extra token multiplies the score by a number below 1, even after the label is already decided. E.g., the score of “shipping address change” would include the probability of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;change&lt;/code&gt;, although &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;change&lt;/code&gt; is the only token that can follow &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;address&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;, we only score each label until we know which label it is. More precisely, we score the tokens up to the token where the label branches away from all the other labels. Let \(d_m\) be the position of this token in label \(m\). Then&lt;/p&gt;

\[\text{score}(m) = \prod_{k=1}^{d_m} P(x_k \mid \text{prompt}, x_1, \dots, x_{k-1}).\]

&lt;p&gt;To illustrate, here are the scores of three labels in the running example. To keep the table short, we leave “prompt” out of each probability:&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Label&lt;/th&gt;
        &lt;th&gt;Score&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;shipping address change&lt;/td&gt;
        &lt;td&gt;\(P(\texttt{shipping}) \times P(\texttt{address} \mid \texttt{shipping})\)&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;product defect&lt;/td&gt;
        &lt;td&gt;\(P(\texttt{product})\)&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;two-factor authentication issue&lt;/td&gt;
        &lt;td&gt;\(P(\texttt{two})\)&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

&lt;/div&gt;

&lt;p&gt;“shipping address change” branches away from the other labels at “address”, because “delay” and “damage” also follow &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt;. “product defect” branches away at “product”, because no other label starts with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;product&lt;/code&gt;. And “two-factor authentication issue” branches away at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;two&lt;/code&gt;, because no other label starts with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;two&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Next, we have to convert the scores into probabilities. The 10 scores won’t add up to 1, because at every token, the LLM spreads its probability over all 151,936 tokens in its vocabulary, including tokens that aren’t part of any label, like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;The&lt;/code&gt;. So Quail divides each score by the sum of all 10 scores:&lt;/p&gt;

\[\hat{P}(m) = \frac{\text{score}(m)}{\sum_{m&apos;} \text{score}(m&apos;)}.\]

&lt;p&gt;Here, \(\hat{P}(m)\) is the probability that Quail returns for label \(m\).&lt;/p&gt;

&lt;p&gt;Finally, we have to handle the scenario where one label is the start of another label. The running example has no such labels. Hypothetically, imagine that we had both “refund” and “refund request” as labels. After &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt;, the LLM can stop, which means the answer is “refund”, or it can continue with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;request&lt;/code&gt;. So the score of “refund” is the probability of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; times the probability of not continuing with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;request&lt;/code&gt;:&lt;/p&gt;

\[\text{score}(\text{refund}) = P(\texttt{refund} \mid \text{prompt}) \times \bigl(1 - P(\texttt{request} \mid \text{prompt}, \texttt{refund})\bigr).\]

&lt;p&gt;Otherwise, the probability of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; would count toward both labels.&lt;/p&gt;

&lt;p&gt;Now, back to the running example. Like in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt;, we could run the LLM at every trie node that has children, which is 8 nodes. However, similarly to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt;, running the LLM at every such node is not as efficient as it could be. The scores above only need the probabilities at the 3 branching nodes. So &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; runs the LLM at 3 nodes (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt;, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt;), all in the same request as the prompt.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Custom attention kernel.&lt;/strong&gt; Lastly, consider how to execute the algorithm above on the GPU. A simple way is to send 3 requests to the LLM, each containing the same prefix (prompt and label descriptions) followed by a different trie node to compute the probability of. Sure, the KV of the prefix may be shared across the 3 requests, but each request would still read the KV of the whole prefix from GPU memory. Instead, Quail sends all 3 nodes in &lt;em&gt;one&lt;/em&gt; request to the LLM, using a custom tree attention kernel.&lt;sup&gt;&lt;a href=&quot;#note-3&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; Each node attends only to the prompt and to the nodes on its own path, as shown in &lt;a href=&quot;#figure-4&quot;&gt;Figure 4&lt;/a&gt;.&lt;/p&gt;

&lt;figure id=&quot;figure-4&quot; style=&quot;width: min(29.7rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/trie-tree-mask.svg&quot; alt=&quot;The attention mask of trie_tree.&quot; /&gt;
  &lt;figcaption&gt;Figure 4. The attention mask of the 3 trie nodes that &lt;code&gt;trie_tree&lt;/code&gt; runs for one document. Each row is a trie node, and a green cell means that the trie node attends to the token in the column.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Overall, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; always processes at least as many tokens as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt;, because it runs the LLM at the branching nodes of every label. However, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; needs only one request per document.&lt;/p&gt;

&lt;h3 id=&quot;33-letters-score-one-token-per-label&quot;&gt;3.3 letters: Score One Token per Label&lt;/h3&gt;

&lt;p&gt;There’s an even simpler way to get probabilities: turn every label into a single token.&lt;sup&gt;&lt;a href=&quot;#note-4&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; puts a letter before each label in the prompt (“A: billing”, “B: refund request”, and so on) and asks the LLM for the letter instead of the label, as shown on the right side of &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt;. Now Quail needs just one step. At &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, it reads the probabilities of the 10 letters, rescales them so they sum to 1, and picks the label whose letter has the highest probability. So there’s no trie and no extra request.&lt;/p&gt;

&lt;p&gt;The downside is a longer prompt, because every label line now has a letter. Accuracy can also change, because the LLM has to answer with a letter instead of the label itself.&lt;sup&gt;&lt;a href=&quot;#note-5&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h3 id=&quot;34-keeping-the-document-kv-for-later-operators&quot;&gt;3.4 Keeping the Document KV for Later Operators&lt;/h3&gt;

&lt;p&gt;So far, we’ve looked at one &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; call on its own. But real queries often run several AI operators over the same documents. E.g., a query might classify the topic of each message, keep only the refund messages, and then classify how urgent they are. Every operator reads the same message, so it would be wasteful to process the message from scratch each time.&lt;/p&gt;

&lt;p&gt;Instead, Quail stores the KV of each document in its KV cache, so the next operator only has to process its own question, even when the operators have different semantics (e.g., a filter, a classification, and a join). Of course, GPU memory is limited, so Quail can’t keep the KV of every document around forever. When memory fills up, Quail evicts the KV that it expects to save the least work per page.&lt;sup&gt;&lt;a href=&quot;#note-6&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h2 id=&quot;4-pricing-and-picking-a-method&quot;&gt;4. Pricing and Picking a Method&lt;/h2&gt;

&lt;p&gt;So which method should Quail use? In short, when &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; can run, it is usually the fastest, assuming that Quail packs every forward pass full of tokens. But &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; is not always the fastest, and the user may also ask for probabilities, which &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; can’t give. So here, we’ll walk through the cost of each method, to see where the differences come from.&lt;/p&gt;

&lt;p&gt;Like all cost models in Quail, our cost models for classification are based on &lt;em&gt;speed-of-light (SoL) estimates&lt;/em&gt;, i.e., the lowest possible time of the work on the GPU. Our &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;blog post on AI-powered filters&lt;/a&gt; explains SoL estimates in detail, and we reuse its notation: \(N\) is the number of documents, \(\ell_i\) is the number of tokens in document \(i\), and \(K\) is the number of forward passes.&lt;/p&gt;

&lt;h3 id=&quot;41-workload-and-notation&quot;&gt;4.1 Workload and Notation&lt;/h3&gt;

&lt;details class=&quot;collapsible-section proof-card&quot;&gt;
  &lt;summary id=&quot;table-1&quot;&gt;Table 1. Symbols in the classification cost model.&lt;/summary&gt;

  &lt;div class=&quot;table-wrap&quot;&gt;

    &lt;table&gt;
      &lt;thead&gt;
        &lt;tr&gt;
          &lt;th&gt;Symbol&lt;/th&gt;
          &lt;th&gt;Meaning&lt;/th&gt;
        &lt;/tr&gt;
      &lt;/thead&gt;
      &lt;tbody&gt;
        &lt;tr&gt;
          &lt;td&gt;\(N\), \(\ell_i\), \(L_1\)&lt;/td&gt;
          &lt;td&gt;Number of documents, tokens in document \(i\), and total document tokens&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(q_{\text{pre}}\), \(q_{\text{tail}}\)&lt;/td&gt;
          &lt;td&gt;Tokens in the preamble and in the instruction (question and label list)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(a\)&lt;/td&gt;
          &lt;td&gt;Expected answer positions per document&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(r_i\), \(n_{\text{tok}}\)&lt;/td&gt;
          &lt;td&gt;Tokens in the request for document \(i\), and in all requests&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(C\), \(K\)&lt;/td&gt;
          &lt;td&gt;Token budget of one forward pass, and number of forward passes&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(M\), \(c_m\), \(b\)&lt;/td&gt;
          &lt;td&gt;Number of labels, branching nodes on the path of label \(m\) up to the one that decides it, and expected number of requests of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; per document&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(B_{\text{kv}}\), \(R\)&lt;/td&gt;
          &lt;td&gt;KV bytes per token, and saved KV tokens that the answer positions read&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(d_{\text{model}}\), \(\text{vocab}\)&lt;/td&gt;
          &lt;td&gt;Width of the hidden state, and number of tokens in the vocabulary&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(V_{\text{out}}\)&lt;/td&gt;
          &lt;td&gt;Rows of the output head that Quail computes at each answer position&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
          &lt;td&gt;\(T_{\text{head}}\), \(T_{\text{classify}}\)&lt;/td&gt;
          &lt;td&gt;Estimated latency of the output head, and of one classification&lt;/td&gt;
        &lt;/tr&gt;
      &lt;/tbody&gt;
    &lt;/table&gt;

  &lt;/div&gt;
&lt;/details&gt;

&lt;p&gt;Let’s start by setting up some notation. Recall from Section 3 that Quail sends the LLM one &lt;em&gt;request&lt;/em&gt; per document. The request contains the &lt;em&gt;preamble&lt;/em&gt; (the text &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;DOCUMENT:&lt;/code&gt;), the document, the &lt;em&gt;instruction&lt;/em&gt; (the question and the list of labels), and then the answer positions. An &lt;em&gt;answer position&lt;/em&gt; is a token after the instruction where Quail reads the scores that the LLM gives to the next token, like the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt; token.&lt;/p&gt;

&lt;p&gt;Let \(q_{\text{pre}}\) and \(q_{\text{tail}}\) be the number of tokens in the preamble and the instruction. Let \(a\) be the expected number of answer positions per document, and let \(s\) be the expected number of answer positions where Quail reads scores. Then the request length, the total number of tokens, and the number of forward passes are&lt;/p&gt;

\[\begin{aligned}
r_i &amp;amp;= q_{\text{pre}} + \ell_i + q_{\text{tail}} + a, \\
n_{\text{tok}} &amp;amp;= L_1 + N(q_{\text{pre}} + q_{\text{tail}} + a), \\
K &amp;amp;= \left\lceil \frac{n_{\text{tok}}}{C} \right\rceil.
\end{aligned}\]

&lt;p&gt;Here, \(L_1 = \sum_i \ell_i\) is the total number of document tokens, and \(C\) is the maximum number of tokens in one forward pass.&lt;/p&gt;

&lt;p&gt;The three methods only differ in \(q_{\text{tail}}\) and \(a\). &lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt; gives the values for our running example.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-2&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Method&lt;/th&gt;
        &lt;th style=&quot;text-align: right&quot;&gt;\(q_{\text{tail}}\)&lt;/th&gt;
        &lt;th style=&quot;text-align: right&quot;&gt;\(a\)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt;&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;66&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;1.5&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;66&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;3&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;89&lt;/td&gt;
        &lt;td style=&quot;text-align: right&quot;&gt;1&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 2. Instruction length and expected answer positions per document in the running example.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;For projections, attention, and the MLP, we can reuse the formulas from &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;our blog post on AI-powered filters&lt;/a&gt;, plugging in the new \(n_{\text{tok}}\) and \(r_i\). However, classification differs from filters in two ways: (1) each method has a different number of KV reads, i.e., loads a different number of cached KV tokens, and (2) classification needs the whole output head, i.e., scores for the whole vocabulary, not just TRUE and FALSE.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;KV reads.&lt;/strong&gt; Each answer position attends to the prompt, so the GPU has to load the KV of the prompt from HBM. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; load it once per document, since all of their answer positions are in one request. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; loads it once per extra decode step. Step 0 shares the request of the prompt, but every later decode step is a new request. So if \(b\) is the expected number of requests of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; per document (including step 0), the number of KV tokens loaded is&lt;/p&gt;

\[\begin{aligned}
R_{\text{letters}} = R_{\text{tree}} &amp;amp;= \sum_i (q_{\text{pre}} + \ell_i + q_{\text{tail}}), \\
R_{\text{decode}} &amp;amp;\approx (b - 1) \sum_i (q_{\text{pre}} + \ell_i + q_{\text{tail}}).
\end{aligned}\]

&lt;p&gt;Plugging \(R\) into the formula from our blog post on AI-powered filters, attention moves \(B_{\text{attn}} = B_{\text{kv}}(W + R)\) bytes. Here, \(B_{\text{kv}}\) is the number of KV bytes for one token across all layers, and \(W\) is the number of tokens whose new KV attention writes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Output head.&lt;/strong&gt; After the last layer, each answer position has a &lt;em&gt;hidden state&lt;/em&gt;, which is a vector of \(d_{\text{model}}\) values. The LLM turns the hidden state into a score for every token in its vocabulary, using one more matrix multiplication. We call the matrix the &lt;em&gt;output head&lt;/em&gt;.&lt;sup&gt;&lt;a href=&quot;#note-7&quot;&gt;7&lt;/a&gt;&lt;/sup&gt; The output head has one row for each vocabulary token, so the score of a token is just the dot product of the hidden state with the row of the token.&lt;/p&gt;

&lt;p&gt;Let \(V_{\text{out}}\) be the number of rows that Quail computes at each answer position. If every label is a single token (like the letters in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;), Quail only computes the rows of the label tokens. Otherwise, Quail computes all \(\text{vocab}\) rows, which it needs to turn the scores into probabilities. Since the output head is in BF16,&lt;/p&gt;

\[\begin{aligned}
F_{\text{head}} &amp;amp;= 2\,d_{\text{model}}\,V_{\text{out}}\,N a, \\
B_{\text{head}} &amp;amp;= 2\,d_{\text{model}}\,V_{\text{out}}\,K, \\
T_{\text{head}} &amp;amp;= \max\!\left(\frac{F_{\text{head}}}{\Pi_{bf16}}, \frac{B_{\text{head}}}{\beta}\right).
\end{aligned}\]

&lt;p&gt;Here, \(F_{\text{head}}\) counts two FLOPs per value of each computed row at each answer position, and \(B_{\text{head}}\) counts two bytes per value of each computed row in each forward pass. \(\Pi_{bf16}\) and \(\beta\) are the peak BF16 throughput and the HBM bandwidth of the GPU. For Qwen3-4B, \(d_{\text{model}} = 2{,}560\) and \(\text{vocab} = 151{,}936\).&lt;/p&gt;

&lt;p&gt;The SoL estimate of one classification is&lt;/p&gt;

\[T_{\text{classify}} = T_{\text{proj}} + T_{\text{attn}} + T_{\text{mlp}} + T_{\text{head}}.\]

&lt;p&gt;Here, \(T_{\text{proj}}\), \(T_{\text{attn}}\), and \(T_{\text{mlp}}\) are the estimated latencies of the projections, attention, and the MLP.&lt;/p&gt;

&lt;h3 id=&quot;42-cost-of-trie_tree-and-letters&quot;&gt;4.2 Cost of trie_tree and letters&lt;/h3&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; each put all of their answer positions in the same request as the prompt, so they only differ in how many tokens they add to each document.&lt;/p&gt;

&lt;p&gt;For &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;, each answer position is one more token that goes through the whole LLM, which costs about 7.2 billion FLOPs in the projections and the MLP, just like a document token. Since some labels have more than one token, Quail also computes the full output head at each answer position, which adds about 0.78 billion FLOPs. In our running example, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; has 3 answer positions, so it adds about 24 billion FLOPs per document (\(a = 3\) and \(V_{\text{out}} = \text{vocab}\)).&lt;/p&gt;

&lt;p&gt;For &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;, the extra tokens are in the prompt instead. The letters add \(2M + 3\) tokens to the instruction, where \(M\) is the number of labels. Each label line gains 2 tokens for its letter, and the question gains 3 tokens for the words “the letter of”. In our running example, the letters add 23 tokens, which cost about 166 billion FLOPs per document. The output head is cheap, because Quail only computes the 10 rows of the letters (\(a = 1\) and \(V_{\text{out}} = M\)). So in our running example, the extra work of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; is about 7 times the extra work of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;. However, both are small next to the document itself, which costs about 7 trillion FLOPs for a 1,000-token document.&lt;/p&gt;

&lt;h3 id=&quot;43-cost-of-trie_decode&quot;&gt;4.3 Cost of trie_decode&lt;/h3&gt;

&lt;p&gt;Unlike the other two methods, the cost of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; depends on which label the LLM picks, because some labels need more decode steps than others. Quail doesn’t know the labels before it runs the query, so the cost model assumes that every label is equally likely.&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Each branching node on the path of a label costs one request. So if \(c_m\) is the number of branching nodes on the path of label \(m\), up to and including the one that decides the label, the expected number of requests per document is&lt;/p&gt;

\[b = \mathbb{E}[\text{requests per document}] = \frac{1}{M} \sum_{m=1}^{M} c_m.\]

&lt;p&gt;In our running example, 5 labels are decided right at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt;, and the other 5 need one more branching node (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt;), as shown in &lt;a href=&quot;#figure-3&quot;&gt;Figure 3&lt;/a&gt;. So&lt;/p&gt;

\[b = \frac{5 \cdot 1 + 5 \cdot 2}{10} = 1.5.\]

&lt;p&gt;In our running example, each decode step adds a single token after the prompt: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt; in step 0, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;shipping&lt;/code&gt; in step 1. So &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; runs the LLM on 1.5 tokens per document on average, beyond the prompt. Like for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;, each token costs about 8 billion FLOPs (7.2 billion in the projections and the MLP, plus 0.78 billion for the full output head). So &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; adds only about 12 billion FLOPs per document, compared with about 24 billion for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Note that the estimate assumes optimal packing, i.e., that Quail packs many documents together so that each forward pass saturates the GPU.&lt;sup&gt;&lt;a href=&quot;#note-9&quot;&gt;9&lt;/a&gt;&lt;/sup&gt; With only a few documents, a later decode step can run in a nearly empty forward pass, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; costs more than the estimate.&lt;/p&gt;

&lt;h3 id=&quot;44-picking-a-method&quot;&gt;4.4 Picking a Method&lt;/h3&gt;

&lt;p&gt;Putting it all together, the optimizer computes \(T_{\text{classify}}\) for every method that can run, and it picks the method with the lowest estimate.&lt;/p&gt;

&lt;p&gt;In our running example, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; wins, because it runs the LLM on the fewest extra tokens per document: 1.5, compared with 3 for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; and 24 for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt; only wins when there are a few labels that share a long start, e.g., 2 labels that differ only in their last word, since both trie methods have to run the LLM on every shared token. And &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; mostly wins when the query asks for probabilities, since &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; can’t run then.&lt;/p&gt;

&lt;p&gt;Note that the differences are small, between 2% and 11% for 1,000 documents of 100 tokens, because all three methods have to prefill the preamble, the document, and the instruction, and the prefill is most of \(T_{\text{classify}}\).&lt;/p&gt;

&lt;h3 id=&quot;45-experiments-on-quail-b&quot;&gt;4.5 Experiments on QUAIL-B&lt;/h3&gt;

&lt;p&gt;To see how all of this plays out end to end, we added 12 classification queries to &lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt;, our benchmark for AI-SQL. The queries span five datasets (movie reviews, adverse drug event reports, fact-checking claims, legal citations, and agent traces), with 4 to 27 labels for each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; operator. Some queries have only one operator (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt;); other queries also have filters, joins, or multiple different &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; operators.&lt;/p&gt;

&lt;p&gt;We ran each query with Qwen3-4B-fp8 on one H100, at scale factor 0.5, on both Quail and a vLLM baseline (v0.26.0, with prefix caching). In the vLLM baseline, the LLM generates the label, and then we parse it. &lt;a href=&quot;#figure-6&quot;&gt;Figure 6&lt;/a&gt; shows the results.&lt;/p&gt;

&lt;figure id=&quot;figure-6&quot; style=&quot;width: min(42rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/quailb-results.svg&quot; alt=&quot;Input tokens per second, KV regret, and cost per query for Quail and a vLLM baseline on the 12 QUAIL-B classification queries.&quot; /&gt;
  &lt;figcaption&gt;Figure 6. QUAIL-B classification queries with Qwen3-4B-fp8 on one H100, at scale factor 0.5. KV regret is the percentage of the tokens an engine computes that it already computed once before, e.g., because it evicted their KV.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Quail is faster on all 12 queries: 2.1x faster in total, and 1.8x faster per query (geometric mean).&lt;/strong&gt; The biggest wins come from queries that read the same documents more than once. E.g., AGENT-5 has three different &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt; operators on the same agent traces. vLLM’s prefix cache evicts the least recently used KV, so by the time the next operator reads a trace, its KV is usually gone and vLLM has to recompute it. On AGENT-5, 64% of the tokens that vLLM computes are recomputations. Quail instead keeps the KV of the trace around for the later operators (&lt;a href=&quot;#34-keeping-the-document-kv-for-later-operators&quot;&gt;Section 3.4&lt;/a&gt;). Quail also runs within 1.8x to 2.8x of its SoL estimate on every query.&lt;/p&gt;

&lt;h2 id=&quot;5-aside-decision-models&quot;&gt;5. Aside: Decision Models&lt;/h2&gt;

&lt;p&gt;Last month, TypeSafe AI released &lt;a href=&quot;https://typesafe.ai/blog/introducing-system-one-models-and-jev&quot;&gt;Jev&lt;/a&gt;, a &lt;em&gt;decision model&lt;/em&gt;. Instead of generating text, Jev answers a typed question (e.g., yes or no, or pick one of a few labels) and returns a probability for each answer. Jev took the community by storm, and open follow-ups came out within weeks: &lt;a href=&quot;https://github.com/jaredpalmer/kev&quot;&gt;Kev&lt;/a&gt; from Jared Palmer, &lt;a href=&quot;https://blog.cloudflare.com/clef-decision-models/&quot;&gt;Clef&lt;/a&gt; from Cloudflare, and the &lt;a href=&quot;https://vllm-sr.ai/blog/decision-models/&quot;&gt;Decision models&lt;/a&gt; from the vLLM Semantic Router team. Databricks even &lt;a href=&quot;https://www.databricks.com/blog/running-open-jev-sql-databricks&quot;&gt;runs open decision models in SQL&lt;/a&gt;. Quail supports one of the open models, &lt;a href=&quot;https://huggingface.co/vllm-sr/Decision-2.0-Kai-0.6B&quot;&gt;Decision 2.0 Kai 0.6B&lt;/a&gt;, as an alternative to the three methods above.&lt;/p&gt;

&lt;p&gt;Decision 2.0 Kai is a fine-tuned version of Qwen3-0.6B. It never generates text; it just returns a probability for each answer, in one forward pass.&lt;/p&gt;

&lt;p&gt;Let’s briefly walk through how it works, as shown in &lt;a href=&quot;#figure-5&quot;&gt;Figure 5&lt;/a&gt;. Instead of listing the labels in the instruction, Quail writes each label into its own &lt;em&gt;option block&lt;/em&gt; after the document. The LLM runs on the whole prompt as usual, and each token comes out of the last layer as a hidden state. Then, an extra &lt;em&gt;decision head&lt;/em&gt; reads the hidden state \(h_m\) of the last token of each option block \(m\), and the hidden state \(q\) of the last token of the prompt.&lt;/p&gt;

&lt;figure id=&quot;figure-5&quot; style=&quot;width: min(42.9rem, calc(100vw - 3rem));&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-classification/decision-head.svg&quot; alt=&quot;The prompt of Decision 2.0 Kai and its decision head.&quot; /&gt;
  &lt;figcaption&gt;Figure 5. The prompt of Decision 2.0 Kai for our running example. The decision head reads the hidden states at the dashed tokens and returns a probability for each label.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;The decision head first rescales \(h_m\) and \(q\) so that their values are on a similar scale, giving \(\bar{h}_m\) and \(\bar{q}\). Then, it computes a score \(s_m\) for each label \(m\), and a softmax turns the scores into probabilities:&lt;/p&gt;

\[\begin{aligned}
s_m &amp;amp;= \frac{(W_h \bar{h}_m) \cdot (W_q \bar{q})}{\sqrt{d}} + w \cdot \operatorname{GELU}\!\left(U_h \bar{h}_m + U_q \bar{q} + b\right), \\
P(m) &amp;amp;= \frac{e^{s_m}}{\sum_{j=1}^{M} e^{s_j}}.
\end{aligned}\]

&lt;p&gt;The first term is a dot product, like an attention score, and the second is a tiny neural network. \(W_h\), \(W_q\), \(U_h\), \(U_q\), \(w\), and \(b\) are weights learned during fine-tuning. The \(W\)s and \(U\)s are \(d \times d_{\text{model}}\) matrices that shrink the hidden states down to \(d\) values, and \(w\) and \(b\) are vectors of \(d\) values that turn the GELU output into a scalar.&lt;sup&gt;&lt;a href=&quot;#note-10&quot;&gt;10&lt;/a&gt;&lt;/sup&gt; For Decision 2.0 Kai, the whole decision head has about 1 million weights, compared with about 600 million in the LLM, so it costs almost nothing.&lt;/p&gt;

&lt;p&gt;Like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;, Decision 2.0 Kai gives probabilities in one request. But &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; has to score every token in the vocabulary at each answer position, while Decision 2.0 Kai only scores the \(M\) labels.&lt;/p&gt;

&lt;p&gt;One might notice that the option blocks add a lot of tokens: about 16 tokens per label, or about 190 tokens per document in our running example, compared with 24 for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;. But the extra tokens (more specifically, the trained decision head) buy a lot of accuracy. We ran Decision 2.0 Kai and the base Qwen3-0.6B on the 12 classification queries in &lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt;, on one H100. Decision 2.0 Kai agrees with the reference labels on 63.1% of documents on average, compared with just 39.0% for Qwen3-0.6B! We are really excited about these decision models and believe we can fine-tune our own decision models for AI-SQL tasks (potentially even-smaller models).&lt;/p&gt;

&lt;h2 id=&quot;6-conclusion&quot;&gt;6. Conclusion&lt;/h2&gt;

&lt;p&gt;In this article, we walked through the many ways to execute &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.CLASSIFY&lt;/code&gt;. Since Quail runs the LLM inside its own inference engine, it can restrict the LLM to the labels: it can greedily generate the label (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt;), score all labels at once (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;), or score one letter per label (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;letters&lt;/code&gt;), and it picks whichever is cheapest for the given LLM and labels. Most of the time, the answer is a trie: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; by default, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; when the query asks for probabilities.&lt;/p&gt;

&lt;p&gt;Next, we’re working on supporting bigger decision models (built on hybrid architectures) and new operators like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AI.EXTRACT&lt;/code&gt;. Stay tuned for more exciting stuff! In the meantime, try out &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt;, and if you like what we’re building, give it a star on GitHub!&lt;/p&gt;

&lt;h2 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h2&gt;

&lt;p&gt;We thank &lt;a href=&quot;https://modal.com/&quot;&gt;Modal&lt;/a&gt; for sponsoring the compute used in
this research.&lt;/p&gt;

&lt;h2 id=&quot;notes&quot;&gt;Notes&lt;/h2&gt;

&lt;p&gt;&lt;span id=&quot;note-1&quot;&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/span&gt; Prior open-source LLM-powered data processing systems from academia, such as &lt;a href=&quot;https://arxiv.org/abs/2410.12189&quot;&gt;DocETL&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/2407.11418&quot;&gt;LOTUS&lt;/a&gt;, and &lt;a href=&quot;https://arxiv.org/abs/2405.14696&quot;&gt;Palimpzest&lt;/a&gt;, also have the LLM generate the answer token by token.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-2&quot;&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/span&gt; Moreover, we also have to use one of the other methods if one label is an exact prefix of another label. E.g., suppose the labels are “refund” and “refund request”. After &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;refund&lt;/code&gt;, the label could already be complete, or it could continue with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;request&lt;/code&gt;. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; picks the next token, but we do not define a token for stopping. So &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; can never pick the label “refund”.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-3&quot;&gt;&lt;strong&gt;3.&lt;/strong&gt;&lt;/span&gt; Our &lt;a href=&quot;/blog/introducing-quail/#324-quail-uses-specialized-inference-programs-for-ai-sql&quot;&gt;blog post introducing Quail&lt;/a&gt; describes the tree attention kernels for AI joins. In a join, many partner documents share one anchor document. In &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt;, many trie nodes share one prompt.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-4&quot;&gt;&lt;strong&gt;4.&lt;/strong&gt;&lt;/span&gt; The structured mode of DiffusionGemma in vLLM uses the same idea. It accepts only one-token choices, and it maps longer options to letters, e.g., “moderation_spam” to “A” (&lt;a href=&quot;https://github.com/vllm-project/vllm/pull/57250&quot;&gt;vLLM PR #57250&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-5&quot;&gt;&lt;strong&gt;5.&lt;/strong&gt;&lt;/span&gt; We don’t know which method is the most accurate in general. On two of our &lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt; queries, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_tree&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; agree with the reference labels about equally often (69.0% vs. 69.1% on BIO-5, and within 0.4 points on each of the three classifications in AGENT-5). For the purposes of the post, we assume that all three methods are about equally accurate, and we expect strong open LLMs to make all three good enough for most queries.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-6&quot;&gt;&lt;strong&gt;6.&lt;/strong&gt;&lt;/span&gt; Quail estimates the savings of a page as the probability that a later operator reads the document, times the time to process the document again, divided by the number of pages of the document. See the &lt;a href=&quot;https://fsdatalab.github.io/quail/docs/architecture/kv#kv-retention&quot;&gt;Quail documentation on KV retention&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-7&quot;&gt;&lt;strong&gt;7.&lt;/strong&gt;&lt;/span&gt; It’s fine for &lt;a href=&quot;/blog/ai-filter-cost-estimates/&quot;&gt;our blog post on AI-powered filters&lt;/a&gt; to leave the cost of the output head out of its cost model, because a filter only needs the rows for TRUE and FALSE, at most 8 rows when we count spellings like True, instead of all 151,936 rows. The 8 rows cost about 41,000 FLOPs per document, while each document token takes about 7.2 billion FLOPs in the projections and the MLP.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-8&quot;&gt;&lt;strong&gt;8.&lt;/strong&gt;&lt;/span&gt; The real labels are usually not equally likely, so the estimate of \(b\) can be too high or too low. E.g., suppose most documents get a label that is decided at the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ANSWER:&lt;/code&gt; token. Then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; sends fewer requests than the estimate, so the SoL estimate of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trie_decode&lt;/code&gt; is too high.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-9&quot;&gt;&lt;strong&gt;9.&lt;/strong&gt;&lt;/span&gt; This is pretty different from online serving engines like vLLM. vLLM cares about the latency of each request, so it runs decode steps as soon as it can, even in mostly empty forward passes. Quail doesn’t care about the latency of each individual document, just the latency of the whole query, so it can always fill the forward pass.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-10&quot;&gt;&lt;strong&gt;10.&lt;/strong&gt;&lt;/span&gt; We found the decision head described in &lt;a href=&quot;https://huggingface.co/vllm-sr/Decision-2.0-Kai-0.6B/blob/881bee413681d80ebeac86afcda8b4138dae516e/decision2/_vendor/dev2model/decision_model.py&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;decision_model.py&lt;/code&gt;&lt;/a&gt;, the code that ships with the model on Hugging Face.&lt;/p&gt;

&lt;h2 id=&quot;cite-this-article&quot;&gt;Cite this article&lt;/h2&gt;

&lt;div class=&quot;bibtex-block&quot;&gt;
  &lt;p&gt;&lt;button class=&quot;copy-bibtex&quot; type=&quot;button&quot;&gt;Copy BibTeX&lt;/button&gt;&lt;/p&gt;

  &lt;div class=&quot;language-bibtex highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nc&quot;&gt;@misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;shankar2026aiclassify&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;title&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{To Classify, Try a Trie Tree}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;author&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Shankar, Shreya and Frye, Charles}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;year&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{2026}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;month&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;oct&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;url&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{https://fsdatalab.github.io/blog/ai-classification/}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

&lt;/div&gt;
</content>
  </entry>
  <entry>
    <title>How to Cost Your AI-Powered Filters</title>
    <link href="https://fsdatalab.github.io/blog/ai-filter-cost-estimates/" rel="alternate" type="text/html"/>
    <id>https://fsdatalab.github.io/blog/ai-filter-cost-estimates/</id>
    <published>2026-10-01T00:00:00+00:00</published>
    <updated>2026-10-01T00:00:00+00:00</updated>
    <author><name>Arnav Dhariya</name><uri>https://arnavdhariya.github.io/</uri></author>
    <author><name>Shreya Shankar</name><uri>https://www.sh-reya.com/</uri></author>
    <summary>How fast could your AI-powered filters run? We do the math for an H100 and show how filter order changes the answer.</summary>
    <content type="html">&lt;aside class=&quot;tldr&quot;&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; How fast could an AI-SQL query run on a given LLM and GPU? We walk through how to estimate speed-of-light (SoL) latency for individual filters and conjunctions of filters, providing a baseline for evaluating system performance. SoL estimates power &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt;&apos;s cost models. You can try out our &lt;a href=&quot;#6-filter-playground&quot;&gt;interactive playground&lt;/a&gt; to explore how filter ordering affects estimated latency on Qwen3-4B and an H100.&lt;/aside&gt;

&lt;nav class=&quot;post-toc&quot; aria-label=&quot;Table of contents&quot;&gt;
&lt;strong&gt;Contents&lt;/strong&gt;
&lt;ol&gt;
  &lt;li&gt;&lt;a href=&quot;#1-introduction&quot;&gt;Introduction.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#2-background&quot;&gt;Background.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#21-ai-powered-filters&quot;&gt;AI-powered filters.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#22-gpu-execution&quot;&gt;GPU Execution.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#23-transformer-forward-pass&quot;&gt;Transformer Forward Pass.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#24-roofline-model--speed-of-light&quot;&gt;Roofline Model &amp;amp; Speed of Light.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#3-cost-model-for-one-filter&quot;&gt;Cost Model for One Filter.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#31-workload-and-notation&quot;&gt;Workload and Notation.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#32-projection-cost&quot;&gt;Projection Cost.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#33-attention-cost&quot;&gt;Attention Cost.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#34-mlp-cost&quot;&gt;MLP Cost.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#35-total-cost&quot;&gt;Total Cost.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#36-cost-of-one-imdb-filter&quot;&gt;Cost of One IMDB Filter.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#4-cost-model-for-a-conjunction-of-filters&quot;&gt;Cost Model for a Conjunction of Filters.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#41-kv-reuse-and-hbm-capacity&quot;&gt;KV Reuse and HBM Capacity.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#42-cost-of-a-fixed-filter-order&quot;&gt;Cost of a Fixed Filter Order.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#43-choosing-the-filter-order&quot;&gt;Choosing the Filter Order.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#5-example-imdb-query&quot;&gt;Example: IMDB Query.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#51-filter-ordering&quot;&gt;Filter Ordering.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#52-sol-estimate-for-the-ordered-conjunction&quot;&gt;SoL Estimate for the Ordered Conjunction.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#6-filter-playground&quot;&gt;Filter Playground.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#7-conclusion&quot;&gt;Conclusion.&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/nav&gt;

&lt;h2 id=&quot;1-introduction&quot;&gt;1. Introduction&lt;/h2&gt;

&lt;p&gt;We recently released &lt;a href=&quot;https://fsdatalab.github.io/blog/introducing-quail/&quot;&gt;Quail&lt;/a&gt;&lt;sup&gt;&lt;a href=&quot;#note-1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;, an execution engine for AI-SQL, which extends SQL with functions that call LLMs. To choose between query plans, Quail needs to estimate how long each AI operation will take. But estimating AI operation latency is not straightforward! In this article, we discuss: &lt;strong&gt;How can we estimate the latency of an AI-SQL query on a given LLM and GPU?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One approach is to profile the system by running representative queries and fitting a model to the measurements. But such a profile would have to be redone for every new model, GPU, or workload, and might not even be accurate — which would be a massive headache. Instead, in Quail, we use the roofline model to estimate the latency of an AI operation.&lt;sup&gt;&lt;a href=&quot;#note-2&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; That is, we compute the arithmetic and memory traffic needed for a query, then divide by the GPU’s peak compute throughput and memory bandwidth, to obtain a &lt;em&gt;speed-of-light&lt;/em&gt; (SoL) estimate, or a lower bound on how quickly a query could run based on hardware limits.&lt;/p&gt;

&lt;p&gt;In this article, we’ll walk through how to derive a SoL estimate for AI-SQL queries that only contain filter operators. Well go through:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The GPU and transformer concepts behind our cost model (&lt;a href=&quot;#2-background&quot;&gt;Section 2&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;How to estimate latency for one AI-powered filter (&lt;a href=&quot;#3-cost-model-for-one-filter&quot;&gt;Section 3&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;How to cost a conjunction of filters and choose a filter order (&lt;a href=&quot;#4-cost-model-for-a-conjunction-of-filters&quot;&gt;Section 4&lt;/a&gt;).&lt;/li&gt;
  &lt;li&gt;An example using Qwen3-4B-fp8 on an H100, with an interactive playground to explore different filter orders (&lt;a href=&quot;#5-example-imdb-query&quot;&gt;Sections 5&lt;/a&gt; and &lt;a href=&quot;#6-filter-playground&quot;&gt;6&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;2-background&quot;&gt;2. Background&lt;/h2&gt;

&lt;p&gt;Here, we first define AI-powered filters. We then describe the hardware and model choices used in our examples (H100 GPU and Qwen3-4B), giving an overview of the parts of a transformer forward pass that affect cost. Finally, we introduce the roofline model we use to estimate SoL latency.&lt;/p&gt;

&lt;h3 id=&quot;21-ai-powered-filters&quot;&gt;2.1 AI-powered filters&lt;/h3&gt;

&lt;p&gt;We define AI-powered filters and introduce the single-filter query and the conjunction of filters used throughout the article.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AI-SQL filter.&lt;/strong&gt; A regular SQL filter evaluates a condition such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;price &amp;gt; 10&lt;/code&gt;. An AI-SQL filter instead asks an LLM to decide whether each row or document satisfies a condition written in natural language. To execute an AI-SQL filter, an LLM is invoked on each row or document and returns a single token, true or false.&lt;/p&gt;

&lt;figure class=&quot;figure-filter-query&quot; id=&quot;figure-1&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-filter-cost-estimates/figure-1.svg&quot; alt=&quot;A database instance of four IMDB reviews, an AI_IF conjunction query applying three predicates, and the resulting table of TRUE/FALSE/— outcomes per review.&quot; /&gt;
  &lt;figcaption&gt;Figure 1. An AI-SQL query applies three filters to four example IMDB reviews. The output shows the result of each filter. A dash in the expected output means that the filter was skipped because the review had already failed an earlier filter.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt; displays the general prompt structure for the AI-powered filter query; its structure is similar to that of a SQL query. Within each invocation of the operator there exists a &lt;em&gt;preamble&lt;/em&gt;, “DOCUMENT:\n”, the actual &lt;em&gt;document&lt;/em&gt;, and the &lt;em&gt;filter instruction&lt;/em&gt; “Evaluate TRUE or FALSE for the following question: […]”. The output of each predicate is a single-token output of true or false. Only the surviving documents pass through to subsequent filters. &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt; shows a query with a conjunction of filters: \(F_1\) (mentions a positive aspect) → \(F_2\) (discusses the ending) → \(F_3\) (mentions a named actor), all three run over the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;reviews&lt;/code&gt; table. We use the first predicate as the single AI-powered filter example in &lt;a href=&quot;#3-cost-model-for-one-filter&quot;&gt;Section 3&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The preamble and filter instruction add tokens to each request, so their lengths affect the cost.&lt;/p&gt;

&lt;h3 id=&quot;22-gpu-execution&quot;&gt;2.2 GPU Execution&lt;/h3&gt;

&lt;p&gt;Our examples assume an NVIDIA H100 SXM GPU. Its Hopper architecture introduced FP8 Tensor Cores as part of a design aimed at accelerating transformer models.&lt;sup&gt;&lt;a href=&quot;#note-3&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; &lt;em&gt;FP8&lt;/em&gt; is an eight-bit floating-point number format.&lt;/p&gt;

&lt;figure class=&quot;figure-full figure-readable&quot; id=&quot;gpu&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-filter-cost-estimates/gpu.svg&quot; alt=&quot;H100 memory hierarchy diagram showing HBM3, L2 cache, and an SM with registers, shared memory, and tensor cores.&quot; /&gt;
  &lt;figcaption&gt;Figure 2. H100 memory layout and forward-pass data movement.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;a href=&quot;#gpu&quot;&gt;Figure 2&lt;/a&gt; shows the parts of the H100 that affect our cost model. &lt;em&gt;High-bandwidth memory (HBM)&lt;/em&gt; is the GPU’s main memory. It stores the model’s weight matrices, embedding table, and &lt;em&gt;key-value (KV) cache&lt;/em&gt;, which holds attention vectors from previously processed tokens for reuse. We explain the vectors in Section 2.3. Data read from HBM passes through the shared L2 cache before being sent to one of the GPU’s 132 &lt;em&gt;streaming multiprocessors (SMs)&lt;/em&gt;. Within each SM, registers and the combined shared memory and L1 cache hold small tiles of data close to the Tensor Cores. Each SM has four Tensor Cores, which perform the matrix multiplications used by the model. The H100 has a 50 MB L2 cache shared by all of its SMs.&lt;/p&gt;

&lt;p&gt;Our cost model tracks two hardware costs: arithmetic and HBM traffic. For arithmetic, it divides the number of &lt;em&gt;floating-point operations (FLOPs)&lt;/em&gt; by the Tensor Core throughput. For HBM traffic, it divides the number of bytes read from or written to HBM by the HBM bandwidth. The model does not account for each intermediate cache level separately.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#table-1&quot;&gt;Table 1&lt;/a&gt; maps the H100 specifications to the symbols used in our equations. &lt;em&gt;Arithmetic throughput&lt;/em&gt; is the number of FLOPs that the GPU can perform per second, and &lt;em&gt;memory bandwidth&lt;/em&gt; is the number of bytes that it can transfer from HBM per second. \(\Pi_{bf16}\) and \(\Pi_{fp8}\) are the peak arithmetic throughput rates for &lt;em&gt;BF16&lt;/em&gt; (a 16-bit floating-point number format) and FP8 operations, while \(\beta\) is the peak HBM bandwidth. We use the FP8 rate for the projection and MLP matrix multiplications, and the BF16 rate for attention in our FlashAttention&lt;sup&gt;&lt;a href=&quot;#note-4&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; setup.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-1&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Symbol&lt;/th&gt;
        &lt;th&gt;Value&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;\(\Pi_{bf16}\)&lt;/td&gt;
        &lt;td&gt;\(989.5\times10^{12}\) FLOP/s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(\Pi_{fp8}\)&lt;/td&gt;
        &lt;td&gt;\(1.979\times10^{15}\) FLOP/s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(\beta\)&lt;/td&gt;
        &lt;td&gt;\(3.35\times10^{12}\) bytes/s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;Capacity&lt;/td&gt;
        &lt;td&gt;80 GB&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 1. H100 SXM hardware constants.&lt;/p&gt;
&lt;/div&gt;

&lt;h3 id=&quot;23-transformer-forward-pass&quot;&gt;2.3 Transformer Forward Pass&lt;/h3&gt;

&lt;p&gt;A &lt;em&gt;forward pass&lt;/em&gt; processes input tokens through the model’s layers. Each filter processes its input and predicts one token, true or false. Within each layer, &lt;em&gt;projections&lt;/em&gt; transform token vectors through matrix multiplication, &lt;em&gt;attention&lt;/em&gt; combines information from the current and earlier tokens, and a &lt;em&gt;multilayer perceptron (MLP)&lt;/em&gt; applies further transformations to each token independently. Our cost model counts the arithmetic and memory traffic of each component within every layer. We use Qwen3-4B FP8&lt;sup&gt;&lt;a href=&quot;#note-5&quot;&gt;5&lt;/a&gt;&lt;/sup&gt; in our examples.&lt;sup&gt;&lt;a href=&quot;#note-6&quot;&gt;6&lt;/a&gt;&lt;/sup&gt; The loaded model occupies about 4.5 GB in HBM.&lt;/p&gt;

&lt;p&gt;Before the first layer, the model uses an &lt;em&gt;embedding table&lt;/em&gt; to map each token ID to a &lt;em&gt;hidden state&lt;/em&gt;, a vector of 2,560 values in Qwen3-4B. Each layer transforms the hidden state before passing it to the next layer.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#model&quot;&gt;Figure 3&lt;/a&gt; shows the execution order within one Qwen3-4B layer. The model first computes query (Q), key (K), and value (V) projections, then performs attention and an output projection. The MLP follows.&lt;/p&gt;

&lt;figure class=&quot;figure-full figure-readable&quot; id=&quot;model&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-filter-cost-estimates/model.svg&quot; alt=&quot;Diagram of a token passing through one Qwen3-4B layer: QKV projection, attention with a KV cache, output projection, and the gate/up/SwiGLU/down MLP block.&quot; /&gt;
  &lt;figcaption&gt;Figure 3. A token passes through Q, K, and V projections, attention, an output projection, and the MLP. Attention also reads the K and V vectors of earlier tokens.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Projections.&lt;/strong&gt; The model’s Q, K, and V projection matrices transform each token’s hidden state into Q, K, and V vectors used by attention. After attention, the &lt;em&gt;output projection&lt;/em&gt; uses another matrix multiplication to produce a vector with the same width as the hidden state. Fortunately, projections can be highly parallelized: each projection applies the same matrix to every token independently, so the GPU can process many tokens at once. Processing more tokens together amortizes the cost of reading the weights from HBM, since the GPU can reuse the same weights across many tokens.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Attention.&lt;/strong&gt; Attention combines the V vectors of the current token and earlier tokens. Comparing the current token’s Q vector with each token’s K vector determines how much each V vector contributes. Unfortunately, each token must be compared with itself and all previous tokens, so doubling the number of tokens roughly quadruples the attention arithmetic.&lt;/p&gt;

&lt;p&gt;Models have multiple &lt;em&gt;attention heads&lt;/em&gt;, and the attention calculation above runs separately for each head. Each head uses its own Q vector and computes its result independently, so the GPU can run the heads in parallel.&lt;/p&gt;

&lt;p&gt;To reduce the memory needed for K and V, Qwen3-4B uses &lt;em&gt;grouped query attention (GQA)&lt;/em&gt;.&lt;sup&gt;&lt;a href=&quot;#note-7&quot;&gt;7&lt;/a&gt;&lt;/sup&gt; Its 32 query heads are arranged in eight groups of four, and the heads in each group share the same K and V vectors. Sharing reduces the K and V projection matrices and KV cache to one quarter the size of &lt;em&gt;full multi-head attention&lt;/em&gt;, where each of the 32 heads has its own K and V vectors.&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;In &lt;a href=&quot;#4-cost-model-for-a-conjunction-of-filters&quot;&gt;Section 4&lt;/a&gt;, we will consider queries with multiple filters on the same document and explain how the filters can reuse the document’s cached K and V vectors (&lt;em&gt;KV cache&lt;/em&gt;).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MLP.&lt;/strong&gt; After attention and the output projection, the &lt;em&gt;multilayer perceptron (MLP)&lt;/em&gt; further transforms each token’s hidden state. It first multiplies the hidden state by the &lt;em&gt;gate&lt;/em&gt; and &lt;em&gt;up&lt;/em&gt; matrices to produce two wider vectors. The SwiGLU &lt;em&gt;activation&lt;/em&gt; first transforms the gate vector, then uses it to scale each element of the up vector.&lt;sup&gt;&lt;a href=&quot;#note-9&quot;&gt;9&lt;/a&gt;&lt;/sup&gt; The &lt;em&gt;down&lt;/em&gt; matrix then returns the result to the original hidden-state width. Like projections, the MLP operates on each token independently, so many tokens can run in parallel and share the weights read from HBM. Its arithmetic grows with the number of tokens, rather than the number of token pairs as in attention.&lt;/p&gt;

&lt;p&gt;The resulting hidden states pass to the next transformer layer, which repeats the projections, attention, and MLP. Qwen3-4B has 36 layers. After the final layer, the model uses the same embedding table to convert the final hidden state into scores for output tokens. &lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt; lists the matrix dimensions and parameter counts we need to calculate the arithmetic and memory traffic across all layers.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-2&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Symbol&lt;/th&gt;
        &lt;th&gt;Meaning&lt;/th&gt;
        &lt;th&gt;Value&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;\(P\)&lt;/td&gt;
        &lt;td&gt;Non-embedding parameters&lt;/td&gt;
        &lt;td&gt;\(3.6\times10^{9}\)&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;layers&lt;/td&gt;
        &lt;td&gt;Transformer layers&lt;/td&gt;
        &lt;td&gt;36&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(d_{\text{model}}\)&lt;/td&gt;
        &lt;td&gt;Hidden-state width&lt;/td&gt;
        &lt;td&gt;2560&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(n_{\text{heads}}\)&lt;/td&gt;
        &lt;td&gt;Query heads&lt;/td&gt;
        &lt;td&gt;32&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(n_{\text{kv}}\)&lt;/td&gt;
        &lt;td&gt;Key-value heads&lt;/td&gt;
        &lt;td&gt;8&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(d_{\text{head}}\)&lt;/td&gt;
        &lt;td&gt;Width of each attention head&lt;/td&gt;
        &lt;td&gt;128&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(d_{\text{mlp}}\)&lt;/td&gt;
        &lt;td&gt;MLP intermediate width&lt;/td&gt;
        &lt;td&gt;9728&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;vocab&lt;/td&gt;
        &lt;td&gt;Vocabulary size&lt;/td&gt;
        &lt;td&gt;151936&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 2. Qwen3-4B architecture constants.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;The parameter count \(P\) includes only projection and MLP weights. The embedding table contains \(\text{vocab}\times d_{\text{model}}\) values, where \(\text{vocab}\) is the number of distinct tokens the model can read or output and \(d_{\text{model}}\) is the hidden-state width.&lt;/p&gt;

&lt;h3 id=&quot;24-roofline-model--speed-of-light&quot;&gt;2.4 Roofline Model &amp;amp; Speed of Light&lt;/h3&gt;

&lt;p&gt;For each component of a forward pass, we need to estimate the time spent on arithmetic and memory transfers. The &lt;em&gt;compute time&lt;/em&gt; is the number of FLOPs divided by the GPU’s arithmetic throughput. The &lt;em&gt;memory transfer time&lt;/em&gt; is the number of bytes read from or written to HBM divided by the HBM bandwidth.&lt;/p&gt;

&lt;p&gt;Sometimes arithmetic takes longer; other times, memory transfers take longer. We use the &lt;em&gt;roofline model&lt;/em&gt; to estimate latency from both times. The model assumes that arithmetic and memory transfers overlap, so the component’s latency is the longer of the two:&lt;/p&gt;

\[T = \max\!\left(\frac{\text{FLOPs}}{\Pi},\; \frac{\text{Bytes Moved}}{\beta}\right).\]

&lt;p&gt;Here, \(T\) is the estimated latency, \(\Pi\) is the arithmetic throughput in FLOP/s, and \(\beta\) is the HBM bandwidth in bytes/s. Using the GPU’s peak arithmetic throughput and memory bandwidth gives a &lt;em&gt;speed-of-light (SoL) latency estimate&lt;/em&gt; for the operation. The estimate is a lower bound because an implementation may not sustain the peak rates or overlap arithmetic and memory transfers perfectly.&lt;/p&gt;

&lt;p&gt;The roofline model predates modern LLMs. Williams et al. introduced it in 2009 to relate arithmetic and memory traffic to hardware performance. We learned a lot about applying it to transformer inference from blog posts by Kipply&lt;sup&gt;&lt;a href=&quot;#note-10&quot;&gt;10&lt;/a&gt;&lt;/sup&gt;, Fergus Finn&lt;sup&gt;&lt;a href=&quot;#note-11&quot;&gt;11&lt;/a&gt;&lt;/sup&gt;, Ben Mayer&lt;sup&gt;&lt;a href=&quot;#note-12&quot;&gt;12&lt;/a&gt;&lt;/sup&gt;, and Modal&lt;sup&gt;&lt;a href=&quot;#note-13&quot;&gt;13&lt;/a&gt;&lt;/sup&gt;, which we recommend reading.&lt;/p&gt;

&lt;p&gt;If memory transfers take longer, the operation is &lt;em&gt;memory-bound&lt;/em&gt;. If arithmetic takes longer, it is &lt;em&gt;compute-bound&lt;/em&gt;. To determine which case applies, we compare the operation’s &lt;em&gt;operational intensity&lt;/em&gt;, \(I\), with the GPU’s &lt;em&gt;ridge point&lt;/em&gt;, \(I^*\):&lt;/p&gt;

\[I = \frac{\text{FLOPs}}{\text{Bytes Moved}}, \qquad
I^* = \frac{\Pi}{\beta}.\]

&lt;p&gt;Here, \(I\) is the arithmetic performed per byte transferred from HBM. \(I^*\) is the intensity at which compute time and memory transfer time are equal. Below the ridge point, the operation is memory-bound. Above it, the operation is compute-bound.&lt;/p&gt;

&lt;p&gt;The H100 has different arithmetic throughput rates for FP8 and BF16, so each has a different ridge point. Our projections and MLP use FP8, while attention uses BF16. For the H100 rates in &lt;a href=&quot;#table-1&quot;&gt;Table 1&lt;/a&gt;:&lt;/p&gt;

\[\begin{aligned}
I^* &amp;amp;= \frac{\Pi_{fp8}}{\beta}
     = \frac{1.979\times10^{15}}{3.35\times10^{12}}
     \approx 590.75\ \text{FLOP/byte}, \\
I^*_{bf16} &amp;amp;= \frac{\Pi_{bf16}}{\beta}
     = \frac{989.5\times10^{12}}{3.35\times10^{12}}
     \approx 295.37\ \text{FLOP/byte}.
\end{aligned}\]

&lt;p&gt;Here, \(I^*\) is the FP8 ridge point used for projections and the MLP, and \(I^*_{bf16}\) is the BF16 ridge point used for attention.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#figure-4&quot;&gt;Figure 4&lt;/a&gt; shows how operational intensity limits arithmetic throughput. In the sloped region, HBM bandwidth is the limit: performing more arithmetic for each byte transferred allows higher throughput. Once an operation reaches the ridge point, arithmetic throughput is the limit, so the plot becomes horizontal.&lt;/p&gt;

&lt;figure class=&quot;figure-full figure-readable&quot; id=&quot;figure-4&quot;&gt;
  &lt;img src=&quot;/assets/blog/ai-filter-cost-estimates/roofline.svg&quot; alt=&quot;Log-log roofline plot for an NVIDIA H100 SXM showing the memory-bound and compute-bound regions, the FP8 and BF16 ridge points, and the operating points of the example single filter and conjunction.&quot; /&gt;
  &lt;figcaption&gt;Figure 4. Roofline for an NVIDIA H100 SXM. The BF16 roof applies only to attention; &amp;times; marks the single filter of Section 3, diamond marks the conjunction of Section 5.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;In &lt;a href=&quot;#35-total-cost&quot;&gt;Section 3.5&lt;/a&gt;, we will calculate how many tokens we need to process before arithmetic, rather than memory transfers, determines each component’s latency.&lt;/p&gt;

&lt;h2 id=&quot;3-cost-model-for-one-filter&quot;&gt;3. Cost Model for One Filter&lt;/h2&gt;

&lt;p&gt;We estimate one filter’s latency by counting the arithmetic and memory traffic for projections, attention, and the MLP. We apply the roofline equation to each component and add the resulting latencies. Using one filter from our example query, we will calculate the cost of processing a batch of documents and predicting true or false for each document.&lt;/p&gt;

&lt;h3 id=&quot;31-workload-and-notation&quot;&gt;3.1 Workload and Notation&lt;/h3&gt;

&lt;p&gt;First, we will introduce notation for evaluating one filter on a batch of documents. A &lt;em&gt;request&lt;/em&gt; is the input sent to the model for one document: the preamble, document, and filter instruction, in that order. We will describe the length of each request, the total number of tokens in the batch, and the number of forward passes needed to process them.&lt;/p&gt;

&lt;p&gt;Let \(N\) be the number of documents in a batch and \(\ell_i\) the number of tokens in document \(i\). Let \(q_{\text{pre}}\) be the number of tokens in the preamble and \(q_{\text{tail}}\) the number of tokens in the filter instruction. We define the following quantities for the batch:&lt;/p&gt;

\[\begin{aligned}
L_1 &amp;amp;= \sum_i \ell_i, \\
r_i &amp;amp;= q_{\text{pre}} + \ell_i + q_{\text{tail}}, \\
n_{\text{tok}} &amp;amp;= \sum_i r_i
    = L_1 + N(q_{\text{pre}} + q_{\text{tail}}), \\
L_2 &amp;amp;= \sum_i r_i^2.
\end{aligned}\]

&lt;p&gt;Here, \(L_1\) is the total number of document tokens in the batch. \(r_i\) is the full request length for document \(i\). \(n_{\text{tok}}\) is the total number of request tokens in the batch, including the preamble, document, and filter instruction in every request. \(L_2\) is the sum of the squared request lengths, which &lt;a href=&quot;#33-attention-cost&quot;&gt;Section 3.3&lt;/a&gt; uses to count attention comparisons.&lt;/p&gt;

&lt;p&gt;Given that we estimate costs before executing the query, we can estimate the mean document length from a sample and use the mean in place of each \(\ell_i\). Using individual document lengths gives a tighter estimate, especially for attention, whose arithmetic depends on squared request lengths.&lt;/p&gt;

&lt;p&gt;Then, let \(C\) be the token budget for one forward pass. We make \(C\) as large as possible so the GPU can reuse the weights across more tokens and read them from HBM fewer times. Quail sets \(C\) using the available HBM and the limits of its GPU programs, called &lt;em&gt;kernels&lt;/em&gt;. For our Qwen3-4B example, the limiting constraint is the kernels’ 32-bit indexing. The token budget and the number of forward passes, \(K\), are&lt;/p&gt;

\[\begin{aligned}
C &amp;amp;= \left\lfloor \frac{2^{31}-1}{2\,d_{\text{mlp}}} \right\rfloor, \\
K &amp;amp;= \left\lceil \frac{n_{\text{tok}}}{C} \right\rceil .
\end{aligned}\]

&lt;p&gt;Here, \(d_{\text{mlp}}\) is the MLP intermediate width from &lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt;. Each token’s combined gate and up vectors occupy \(2d_{\text{mlp}}\) elements, so Quail’s 32-bit indexing limit of \(2^{31}-1\) elements determines \(C\). In &lt;a href=&quot;#41-kv-reuse-and-hbm-capacity&quot;&gt;Section 4.1&lt;/a&gt;, when we discuss multiple filters, the batch size will be further limited by the HBM available to retain documents’ KV caches.&lt;/p&gt;

&lt;h3 id=&quot;32-projection-cost&quot;&gt;3.2 Projection Cost&lt;/h3&gt;

&lt;p&gt;Here we derive the FLOPs for Q, K, V, and output projections, their HBM traffic across forward passes, and the projection roofline cost.&lt;/p&gt;

&lt;p&gt;Q, K, V, and output projections are matrix multiplications over each token. Each matrix maps a vector of one width to another, so its parameter count is the product of its input and output widths. Per layer we have the following for each projection:&lt;/p&gt;

\[\begin{aligned}
p_Q &amp;amp;= d_{\text{model}}\,n_{\text{heads}}\,d_{\text{head}}, \\
p_K &amp;amp;= d_{\text{model}}\,n_{kv}\,d_{\text{head}}, \\
p_V &amp;amp;= d_{\text{model}}\,n_{kv}\,d_{\text{head}}, \\
p_O &amp;amp;= n_{\text{heads}}\,d_{\text{head}}\,d_{\text{model}}.
\end{aligned}\]

&lt;p&gt;Here, \(p_Q\), \(p_K\), \(p_V\), and \(p_O\) are the parameter counts of the Q, K, V, and output projection matrices in one layer.&lt;/p&gt;

&lt;p&gt;Thus, in every layer they hold \(2\,d_{\text{model}}\,d_{\text{head}}(n_{\text{heads}}+n_{kv})\) parameters. The totals for parameters, FLOPs, and memory transfer are given below along with the total projection cost.&lt;/p&gt;

\[\begin{aligned}
P_{\text{proj}} &amp;amp;= \text{layers}\cdot 2\,d_{\text{model}}\,d_{\text{head}}(n_{\text{heads}}+n_{kv}), \\
F_{\text{proj}} &amp;amp;= 2\,P_{\text{proj}}\,n_{\text{tok}}, \\
B_{\text{proj}} &amp;amp;= b_w\,P_{\text{proj}}\,K, \\
T_{\text{proj}} &amp;amp;= \max\!\left(\frac{F_{\text{proj}}}{\Pi_{fp8}}, \frac{B_{\text{proj}}}{\beta}\right).
\end{aligned}\]

&lt;p&gt;Here, \(P_{\text{proj}}\) is the number of projection parameters across all layers. \(F_{\text{proj}}\) is the total number of projection FLOPs. For each token, we multiply each weight by an input value and add the product to the output sum, counting two FLOPs per weight. \(b_w\) is the number of bytes per weight, which is 1 for FP8. \(B_{\text{proj}}\) is the number of projection weight bytes read from HBM across all \(K\) forward passes. \(T_{\text{proj}}\) is the projection latency, which is the larger of compute time and memory transfer time.&lt;/p&gt;

&lt;h3 id=&quot;33-attention-cost&quot;&gt;3.3 Attention Cost&lt;/h3&gt;

&lt;p&gt;We now count attention’s arithmetic and memory traffic and use the roofline equation to estimate its latency.&lt;/p&gt;

&lt;p&gt;Attention is different from the projections, as tokens here interact with each other. Each token attends to itself and to every earlier token in its own request. A request of \(r_i\) tokens makes \(1 + 2 + \dots + r_i - 1 + r_i\) comparisons, so across the batch&lt;/p&gt;

\[A = \sum_i \frac{r_i(r_i+1)}{2} = \frac{n_{\text{tok}} + L_2}{2},\]

&lt;p&gt;Here, \(A\) is the total number of attention comparisons across the batch. \(n_{\text{tok}} = \sum_i r_i\) is the total number of request tokens, and \(L_2 = \sum_i r_i^2\) is the sum of the squared request lengths defined in &lt;a href=&quot;#31-workload-and-notation&quot;&gt;Section 3.1&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;One comparison, for one head in one layer, costs \(2\,d_{\text{head}}\) FLOPs for the dot product of Q and K vectors, because we do a multiply and addition per number, and another \(2\,d_{\text{head}}\) for the weighted sum over V vectors. Repeating across all \(n_{\text{heads}}\) and all layers gives,&lt;/p&gt;

\[F_{\text{attn}} = 4\,n_{\text{heads}}\,d_{\text{head}}\,\text{layers}\cdot A.\]

&lt;p&gt;Here, \(F_{\text{attn}}\) is the total number of attention FLOPs across all heads and layers.&lt;/p&gt;

&lt;p&gt;Unlike projections, attention has no weights of its own to read. Instead, attention uses the K and V vectors (KV) produced by the projections. For large batches, the KV vectors do not all fit in the GPU’s L2 cache, so they are stored in a temporary buffer in HBM.&lt;/p&gt;

&lt;p&gt;Let \(B_{\text{kv}}\) be the bytes needed to store one token’s KV vectors across all layers, and \(b_{kv}\) the bytes per vector element (2 for BF16):&lt;/p&gt;

\[B_{\text{kv}} = 2\,b_{kv}\,\text{layers}\,n_{kv}\,d_{\text{head}},\]

&lt;p&gt;The factor 2 accounts for one K vector and one V vector. Even though Qwen3-4B has 32 query heads, it stores only eight pairs of K and V vectors per token per layer, because every four query heads share a pair.&lt;/p&gt;

&lt;p&gt;For a single filter, we write KV vectors for all \(n_{\text{tok}}\) input tokens. Our estimate of attention’s memory traffic, \(B_{\text{attn}}\), is therefore&lt;/p&gt;

\[B_{\text{attn}} = B_{\text{kv}}\,n_{\text{tok}}.\]

&lt;p&gt;Let \(W\) count tokens whose new KV vectors we write and \(R\) count tokens whose saved KV vectors we read:&lt;/p&gt;

\[B_{\text{attn}} = B_{\text{kv}}\,(W + R).\]

&lt;p&gt;For a single filter, \(W = n_{\text{tok}}\) and \(R = 0\) because no vectors have been saved by an earlier filter.&lt;sup&gt;&lt;a href=&quot;#note-14&quot;&gt;14&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Now, back to our roofline estimate. Our FlashAttention implementation uses BF16, so we use the H100’s BF16 arithmetic throughput:&lt;/p&gt;

\[T_{\text{attn}} = \max\!\left(\frac{F_{\text{attn}}}{\Pi_{bf16}}, \frac{B_{\text{attn}}}{\beta}\right).\]

&lt;p&gt;Here, \(T_{\text{attn}}\) is the attention latency, which is the larger of compute time and memory transfer time.&lt;/p&gt;

&lt;h3 id=&quot;34-mlp-cost&quot;&gt;3.4 MLP Cost&lt;/h3&gt;

&lt;p&gt;We still need to derive the FLOPs for the gate, up, and down MLP projections, their HBM traffic, and the MLP roofline cost.&lt;/p&gt;

&lt;p&gt;The MLP acts on each token independently, so unlike attention its cost per token does not depend on the other tokens in the request. It has three matrices per layer, and the parameter count of each is the product of its input and output widths. \(W_{\text{gate}}\) and \(W_{\text{up}}\) map \(d_{\text{model}}\) to \(d_{\text{mlp}}\), while \(W_{\text{down}}\) maps \(d_{\text{mlp}}\) back to \(d_{\text{model}}\):&lt;/p&gt;

\[\begin{aligned}
p_{\text{gate}} &amp;amp;= d_{\text{model}}\,d_{\text{mlp}}, \\
p_{\text{up}}   &amp;amp;= d_{\text{model}}\,d_{\text{mlp}}, \\
p_{\text{down}} &amp;amp;= d_{\text{mlp}}\,d_{\text{model}}.
\end{aligned}\]

&lt;p&gt;Here, \(p_{\text{gate}}\), \(p_{\text{up}}\), and \(p_{\text{down}}\) are the parameter counts of the gate, up, and down matrices in one layer.&lt;/p&gt;

&lt;p&gt;Gate and up read the same input; their outputs are combined by SwiGLU (Swish Gated Linear Unit) activation and projected back down. The total parameters in the MLP are,&lt;/p&gt;

\[P_{\text{mlp}} = \text{layers}\cdot(p_{\text{gate}} + p_{\text{up}} + p_{\text{down}}) = 3\,\text{layers}\,d_{\text{model}}\,d_{\text{mlp}}.\]

&lt;p&gt;Here, \(P_{\text{mlp}}\) is the number of MLP parameters across all layers.&lt;/p&gt;

&lt;p&gt;The reasoning for the MLP is the same as it is for the projections: every token is multiplied by every weight once, and the weights are streamed from HBM once per forward pass.&lt;/p&gt;

\[\begin{aligned}
F_{\text{mlp}} &amp;amp;= 2\,P_{\text{mlp}}\,n_{\text{tok}}, \\
B_{\text{mlp}} &amp;amp;= b_w\,P_{\text{mlp}}\,K.
\end{aligned}\]

&lt;p&gt;Here, \(F_{\text{mlp}}\) is the total number of MLP FLOPs. \(B_{\text{mlp}}\) is the number of MLP weight bytes read from HBM across all \(K\) forward passes.&lt;/p&gt;

&lt;p&gt;The MLP matrix multiplications use FP8, so the roofline estimate uses the H100’s FP8 arithmetic throughput:&lt;/p&gt;

\[T_{\text{mlp}} = \max\!\left(\frac{F_{\text{mlp}}}{\Pi_{fp8}}, \frac{B_{\text{mlp}}}{\beta}\right).\]

&lt;p&gt;Here, \(T_{\text{mlp}}\) is the MLP latency, which is the larger of compute time and memory transfer time.&lt;/p&gt;

&lt;h3 id=&quot;35-total-cost&quot;&gt;3.5 Total Cost&lt;/h3&gt;

&lt;p&gt;A forward pass runs the projections, attention, and the MLP in sequence, so the three component times add:&lt;/p&gt;

\[T_{\text{filter}} = T_{\text{proj}} + T_{\text{attn}} + T_{\text{mlp}}.\]

&lt;p&gt;Here, \(T_{\text{filter}}\) is the total latency of one filter.&lt;/p&gt;

&lt;p&gt;We will go through the following questions to give you some intuition for how batch size and request length affect latency.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When do projections and the MLP become compute-bound?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For either projections or the MLP, let \(P\) be the component’s weight count. &lt;a href=&quot;#32-projection-cost&quot;&gt;Section 3.2&lt;/a&gt; and &lt;a href=&quot;#34-mlp-cost&quot;&gt;Section 3.4&lt;/a&gt; give \(2P n_{\text{tok}}\) FLOPs and \(b_w P K\) bytes of weight reads. With FP8 weights, \(b_w = 1\), so the operational intensity, \(I\), is&lt;/p&gt;

\[I = \frac{2P n_{\text{tok}}}{b_w P K}
  = 2\,\frac{n_{\text{tok}}}{K}.\]

&lt;p&gt;The weight count cancels, leaving twice the average number of tokens per forward pass. The factor 2 counts the multiplication and addition performed for each weight and token.&lt;sup&gt;&lt;a href=&quot;#note-15&quot;&gt;15&lt;/a&gt;&lt;/sup&gt; To exceed the H100’s FP8 ridge point of about 591 FLOPs per byte, we need only about 296 tokens per forward pass. Batch-processing workloads typically exceed that threshold, so projections and the MLP are compute-bound.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When does attention dominate latency?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Once the components are compute-bound, we can compare their arithmetic times directly. From the earlier roofline formulas,&lt;/p&gt;

\[\begin{aligned}
T_{\text{proj}} + T_{\text{mlp}}
&amp;amp;\approx \frac{2(P_{\text{proj}}+P_{\text{mlp}})\,n_{\text{tok}}}{\Pi_{fp8}}, \\
T_{\text{attn}}
&amp;amp;\approx \frac{4\,n_{\text{heads}}\,d_{\text{head}}\,\text{layers}\,A}{\Pi_{bf16}}.
\end{aligned}\]

&lt;p&gt;Projection and MLP arithmetic grows with the number of tokens; attention arithmetic grows with the number of token pairs. If every request contains \(r\) tokens, then \(n_{\text{tok}} = Nr\) and \(A = Nr(r+1)/2\). Dividing the attention time by the combined projection and MLP time gives&lt;/p&gt;

\[\begin{aligned}
\frac{T_{\text{attn}}}{T_{\text{proj}}+T_{\text{mlp}}}
&amp;amp;\approx \frac{n_{\text{heads}}\,d_{\text{head}}\,\text{layers}\,(r+1)}
{P_{\text{proj}}+P_{\text{mlp}}}
\cdot \frac{\Pi_{fp8}}{\Pi_{bf16}} \\
&amp;amp;= \frac{r+1}{12{,}320}.
\end{aligned}\]

&lt;p&gt;With our model and hardware specs, projections and MLPs dominate the estimated latency for requests shorter than about 12,300 tokens. Above that length, attention dominates.&lt;/p&gt;

&lt;h3 id=&quot;36-cost-of-one-imdb-filter&quot;&gt;3.6 Cost of One IMDB Filter&lt;/h3&gt;

&lt;p&gt;We can finally estimate the SoL latency of applying \(F_1\), the filter that asks whether a review mentions a positive aspect of the movie, to 5,000 IMDB reviews! This is the first filter in &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt;. We use the H100 specs, along with the following batch and prompt values:&lt;/p&gt;

\[N = 5{,}000, \qquad
\bar{\ell} = 298.8466 \ \text{tokens}, \qquad
q_{\text{pre}} = 2 \ \text{tokens}, \qquad
q_{\text{tail}} = 51 \ \text{tokens}.\]

&lt;p&gt;Here, \(N\) is the number of reviews, \(\bar{\ell}\) is the mean review length, \(q_{\text{pre}}\) is the preamble length, and \(q_{\text{tail}}\) is the length of \(F_1\)’s instruction.&lt;/p&gt;

&lt;p&gt;We approximate every document by the mean length, which gives&lt;/p&gt;

\[L_1 = N\bar{\ell} = 1{,}494{,}233 \ \text{tokens}.\]

&lt;p&gt;&lt;strong&gt;Request and batch sizes.&lt;/strong&gt; Each request contains the preamble, one document, and the filter instruction:&lt;/p&gt;

\[\begin{aligned}
r &amp;amp;= q_{\text{pre}} + \bar{\ell} + q_{\text{tail}}
  = 2 + 298.8466 + 51
  = 351.8466 \ \text{tokens}, \\
n_{\text{tok}} &amp;amp;= N r
   = 5{,}000 \times 351.8466
   = 1{,}759{,}233.
\end{aligned}\]

&lt;p&gt;Here, \(r\) is the common request length under the mean-length approximation. \(n_{\text{tok}}\) is the total number of request tokens across all 5,000 reviews.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Attention comparisons.&lt;/strong&gt; The attention calculation also needs the sum of squared request lengths:&lt;/p&gt;

\[\begin{aligned}
L_2 &amp;amp;= N r^2
    = 5{,}000 \times 351.8466^2
    = 618{,}980{,}149.66, \\
A &amp;amp;= \frac{n_{\text{tok}} + L_2}{2}
    = 310{,}369{,}691.33 .
\end{aligned}\]

&lt;p&gt;Here, \(L_2\) is the sum of the squared request lengths, and \(A\) is the total number of attention comparisons. Using the mean length for every request underestimates attention arithmetic when review lengths vary, because the mean of squared lengths is larger than the square of the mean length.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Forward passes.&lt;/strong&gt; Using the token budget from &lt;a href=&quot;#31-workload-and-notation&quot;&gt;Section 3.1&lt;/a&gt;, \(C = 110{,}376\), processing all 5,000 reviews requires&lt;/p&gt;

\[K = \left\lceil \frac{1{,}759{,}233}{110{,}376} \right\rceil
  = 16 \text{ forward passes}.\]

&lt;p&gt;&lt;strong&gt;Model parameters.&lt;/strong&gt; We substitute the Qwen3-4B dimensions from &lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt;:&lt;/p&gt;

\[\begin{aligned}
d_{\text{model}} &amp;amp;= 2560, &amp;amp;
d_{\text{mlp}} &amp;amp;= 9728, &amp;amp;
d_{\text{head}} &amp;amp;= 128, \\
n_{\text{heads}} &amp;amp;= 32, &amp;amp;
n_{kv} &amp;amp;= 8, &amp;amp;
\text{layers} &amp;amp;= 36.
\end{aligned}\]

\[\begin{aligned}
P_{\text{proj}} &amp;amp;= 36 \cdot 2 \cdot 2560 \cdot 128 \cdot (32+8)
                  = 943{,}718{,}400, \\
P_{\text{mlp}}  &amp;amp;= 3 \cdot 36 \cdot 2560 \cdot 9728
                  = 2{,}689{,}597{,}440 .
\end{aligned}\]

&lt;p&gt;&lt;strong&gt;Projections.&lt;/strong&gt;&lt;/p&gt;

\[\begin{aligned}
F_{\text{proj}}
&amp;amp;= 2 \times 943{,}718{,}400 \times 1{,}759{,}233 \\
&amp;amp;\approx 3.3204\times10^{15}\ \text{FLOPs},\\[0.4em]
B_{\text{proj}}
&amp;amp;= 1 \times 943{,}718{,}400 \times 16 \\
&amp;amp;\approx 1.5099\times10^{10}\ \text{bytes},\\[0.4em]
T_{\text{proj}}
&amp;amp;= \max\!\left(
\frac{3.3204\times10^{15}}{1.979\times10^{15}},
\frac{1.5099\times10^{10}}{3.35\times10^{12}}
\right)\\[0.3em]
&amp;amp;= \max(1.6778,\ 0.0045)
= 1.6778\ \text{s}.
\end{aligned}\]

&lt;p&gt;&lt;strong&gt;Attention.&lt;/strong&gt;&lt;/p&gt;

\[\begin{aligned}
F_{\text{attn}}
&amp;amp;= 4 \times 32 \times 128 \times 36 \times A \\
&amp;amp;= 589{,}824 \times 310{,}369{,}691.33 \\
&amp;amp;\approx 1.8306\times10^{14}\ \text{FLOPs},\\[0.4em]
B_{\text{kv}}
&amp;amp;= 2 \times 2 \times 36 \times 8 \times 128 \\
&amp;amp;= 147{,}456\ \text{bytes/token},\\[0.4em]
B_{\text{attn}}
&amp;amp;= 147{,}456 \times (1{,}759{,}233 + 0) \\
&amp;amp;\approx 2.5941\times10^{11}\ \text{bytes},\\[0.4em]
T_{\text{attn}}
&amp;amp;= \max\!\left(
\frac{1.8306\times10^{14}}{989.5\times10^{12}},
\frac{2.5941\times10^{11}}{3.35\times10^{12}}
\right)\\[0.3em]
&amp;amp;= \max(0.1850,\ 0.0774)
= 0.1850\ \text{s}.
\end{aligned}\]

&lt;p&gt;&lt;strong&gt;MLP.&lt;/strong&gt;&lt;/p&gt;

\[\begin{aligned}
F_{\text{mlp}}
&amp;amp;= 2 \times 2{,}689{,}597{,}440 \times 1{,}759{,}233 \\
&amp;amp;\approx 9.4633\times10^{15}\ \text{FLOPs},\\[0.4em]
B_{\text{mlp}}
&amp;amp;= 1 \times 2{,}689{,}597{,}440 \times 16 \\
&amp;amp;\approx 4.3034\times10^{10}\ \text{bytes},\\[0.4em]
T_{\text{mlp}}
&amp;amp;= \max\!\left(
\frac{9.4633\times10^{15}}{1.979\times10^{15}},
\frac{4.3034\times10^{10}}{3.35\times10^{12}}
\right)\\[0.3em]
&amp;amp;= \max(4.7818,\ 0.0128)
= 4.7818\ \text{s}.
\end{aligned}\]

&lt;p&gt;&lt;strong&gt;Total.&lt;/strong&gt;&lt;/p&gt;

\[\begin{aligned}
T_{\text{filter}}
&amp;amp;= T_{\text{proj}} + T_{\text{attn}} + T_{\text{mlp}} \\
&amp;amp;= 1.6778 + 0.1850 + 4.7818 \\
&amp;amp;= \boxed{6.64\ \text{seconds}}.
\end{aligned}\]

&lt;div class=&quot;table-wrap&quot; id=&quot;table-3&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Component&lt;/th&gt;
        &lt;th&gt;FLOPs&lt;/th&gt;
        &lt;th&gt;Bytes&lt;/th&gt;
        &lt;th&gt;\(T_{\text{compute}}\)&lt;/th&gt;
        &lt;th&gt;\(T_{\text{memory}}\)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;Projections&lt;/td&gt;
        &lt;td&gt;\(3.32\times10^{15}\)&lt;/td&gt;
        &lt;td&gt;\(1.51\times10^{10}\)&lt;/td&gt;
        &lt;td&gt;1.6778 s&lt;/td&gt;
        &lt;td&gt;0.0045 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;Attention&lt;/td&gt;
        &lt;td&gt;\(1.83\times10^{14}\)&lt;/td&gt;
        &lt;td&gt;\(2.59\times10^{11}\)&lt;/td&gt;
        &lt;td&gt;0.1850 s&lt;/td&gt;
        &lt;td&gt;0.0774 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;MLP&lt;/td&gt;
        &lt;td&gt;\(9.46\times10^{15}\)&lt;/td&gt;
        &lt;td&gt;\(4.30\times10^{10}\)&lt;/td&gt;
        &lt;td&gt;4.7818 s&lt;/td&gt;
        &lt;td&gt;0.0128 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;\(1.30\times10^{16}\)&lt;/td&gt;
        &lt;td&gt;\(3.17\times10^{11}\)&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;6.64 s&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;0.09 s&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 3. Estimated cost of applying &lt;em&gt;F&lt;sub&gt;1&lt;/sub&gt;&lt;/em&gt; (mentions a positive aspect) to all 5,000 IMDB reviews on Qwen3-4B and an H100. Every component is compute-bound.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;The estimated SoL latency is 6.64 seconds for all 5,000 reviews, not for one review! So fast!&lt;/p&gt;

&lt;h2 id=&quot;4-cost-model-for-a-conjunction-of-filters&quot;&gt;4. Cost Model for a Conjunction of Filters&lt;/h2&gt;

&lt;p&gt;Now let’s estimate the cost of applying several filters to the same documents. We can skip later filters for a document that fails, and reuse the document’s KV vectors if it passes. First, we will find a batch size that lets us keep the KV vectors in HBM. Then we will compare filter orders using their per-document costs. Finally, we will derive the SoL estimate for the selected order.&lt;/p&gt;

&lt;h3 id=&quot;41-kv-reuse-and-hbm-capacity&quot;&gt;4.1 KV Reuse and HBM Capacity&lt;/h3&gt;

&lt;p&gt;Taking a closer look at &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt;, every filter prompt for a given document contains the same preamble and document. Only the filter instruction changes. We call the shared preamble and document the &lt;em&gt;prefix&lt;/em&gt;. Its average length, \(p\), is&lt;/p&gt;

\[p = q_{\text{pre}} + \bar{\ell}.\]

&lt;p&gt;The first filter computes the prefix’s KV vectors. Later filters reuse them and process only their own instruction tokens. Our cost model assumes the saved vectors remain in HBM until the document fails a filter or passes the final filter. We therefore choose a batch whose prefix KV vectors fit in HBM.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;HBM available for KV.&lt;/strong&gt; The GPU needs space for model weights, temporary intermediate vectors called &lt;em&gt;activations&lt;/em&gt;, and saved KV vectors. We use a rough estimate of peak activation memory.&lt;sup&gt;&lt;a href=&quot;#note-16&quot;&gt;16&lt;/a&gt;&lt;/sup&gt; Inference engines such as &lt;a href=&quot;https://docs.vllm.ai/en/v0.18.2/api/vllm/v1/worker/gpu_worker/#vllm.v1.worker.gpu_worker.Worker.determine_available_memory&quot;&gt;vLLM&lt;/a&gt; instead run a forward pass with dummy inputs to profile peak activation memory before determining how much space remains for KV.&lt;/p&gt;

&lt;p&gt;Quail makes 95% of the H100’s 80 GB available to its memory pool. Let \(M_{\text{KV}}\) be the space left for saved KV vectors after accounting for resident weights, including the embedding table and any extra storage required by FP8 weights, and temporary buffers:&lt;/p&gt;

\[M_{\text{KV}}
= \text{usable HBM} - \text{resident weights} - \text{temporary buffers}.\]

&lt;p&gt;&lt;a href=&quot;#table-4&quot;&gt;Table 4&lt;/a&gt; gives the resulting budget for our example.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-4&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Item&lt;/th&gt;
        &lt;th&gt;Size (GB)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;Usable HBM (95% of 80 GB)&lt;/td&gt;
        &lt;td&gt;76.00&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;− Resident weights&lt;/td&gt;
        &lt;td&gt;4.50&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;− Estimated temporary buffers&lt;/td&gt;
        &lt;td&gt;18.08&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;strong&gt;Left for KV cache&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;53.42&lt;/strong&gt;&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 4. Estimated HBM budget for Qwen3-4B on an H100 SXM.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Batch size in documents.&lt;/strong&gt; &lt;a href=&quot;#33-attention-cost&quot;&gt;Section 3.3&lt;/a&gt; gives \(B_{\text{kv}} = 147{,}456\) bytes per token. With an average prefix length of \(p = 300.8466\) tokens, one document’s prefix KV takes about 44.4 MB. Let \(N_b\) be the maximum number of documents in a batch whose KV vectors fit in HBM. Unlike \(C\), which counts tokens per forward pass, \(N_b\) counts documents. We divide the available KV memory by the bytes needed for one document’s prefix:&lt;/p&gt;

\[N_b = \left\lfloor \frac{M_{\text{KV}}}{B_{\text{kv}}\,p} \right\rfloor
\approx 1{,}200\ \text{documents}.\]

&lt;p&gt;We therefore split the 5,000 reviews into five batches. Because the bound uses the mean document length, it is an estimate rather than a guarantee for every batch.&lt;/p&gt;

&lt;h3 id=&quot;42-cost-of-a-fixed-filter-order&quot;&gt;4.2 Cost of a Fixed Filter Order&lt;/h3&gt;

&lt;p&gt;Suppose a query applies \(m\) filters to \(N\) documents. We write the filter order as \(\pi\), a list of filters. For example, \(\pi=(F_2,F_1,F_3)\) means we evaluate \(F_2\) first, then \(F_1\), then \(F_3\). The symbol \(\pi_j\) denotes the filter in position \(j\), so in this example \(\pi_1=F_2\), \(\pi_2=F_1\), and \(\pi_3=F_3\).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Selectivity&lt;/em&gt;, \(s_i\), is the fraction of documents that pass filter \(F_i\). Let \(N_j\) be the expected number of documents evaluated by the filter in position \(j\). We evaluate that filter only on documents that pass every preceding filter:&lt;sup&gt;&lt;a href=&quot;#note-17&quot;&gt;17&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

\[N_j = N \prod_{k&amp;lt;j} s_{\pi_k}.\]

&lt;p&gt;As in Section 3, we count the tokens processed \(n\), attention comparisons \(A\), tokens whose KV vectors are written \(W\), and tokens whose saved KV vectors are read \(R\). We shorten \(n_{\text{tok}}\) to \(n\) here and use \(n_j\) for the tokens processed by the filter in position \(j\).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First filter.&lt;/strong&gt; Let \(q_i\) be filter \(F_i\)’s instruction length, corresponding to \(q_{\text{tail}}\) in the single-filter model. We write \(q_{\pi_j}\) for the instruction length of the filter in position \(j\). Using document lengths \(\ell_i\) from &lt;a href=&quot;#31-workload-and-notation&quot;&gt;Section 3.1&lt;/a&gt;, the first request for document \(i\) has length \(r_i = q_{\text{pre}} + \ell_i + q_{\pi_1}\). The first filter processes all \(N\) documents:&lt;/p&gt;

\[\begin{aligned}
n_1 &amp;amp;= \sum_{i=1}^{N} r_i, \\
A_1 &amp;amp;= \sum_{i=1}^{N}\frac{r_i(r_i+1)}{2}, \\
W_1 &amp;amp;= n_1, \\
R_1 &amp;amp;= 0.
\end{aligned}\]

&lt;p&gt;Here, \(n_1\) is the number of tokens processed, \(A_1\) is the number of attention comparisons, and \(W_1\) is the number of tokens whose new KV vectors are written. No saved KV vectors are available to read, so \(R_1 = 0\).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Later filters.&lt;/strong&gt; At position \(j&amp;gt;1\), we process only the instruction tokens for the \(N_j\) surviving documents.&lt;sup&gt;&lt;a href=&quot;#note-18&quot;&gt;18&lt;/a&gt;&lt;/sup&gt; Each instruction token attends to the cached prefix of average length \(p\) and to itself and earlier instruction tokens:&lt;/p&gt;

\[\begin{aligned}
n_j &amp;amp;= N_j\,q_{\pi_j}, \\
A_j &amp;amp;= N_j\left[q_{\pi_j}\,p + \frac{q_{\pi_j}(q_{\pi_j}+1)}{2}\right], \\
W_j &amp;amp;= n_j, \\
R_j &amp;amp;= N_j\,p.
\end{aligned}\]

&lt;p&gt;Here, \(n_j\) is the number of instruction tokens processed, \(A_j\) is the number of attention comparisons, \(W_j\) counts tokens whose new KV vectors are written, and \(R_j\) counts tokens whose saved KV vectors are read. In \(A_j\), the first term counts comparisons with the prefix and the second counts comparisons within the instruction.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;From work to cost.&lt;/strong&gt; Write \(T_{\text{filter}}(n,A,W,R)\) for the latency obtained by substituting the workload counts into the equations from &lt;a href=&quot;#3-cost-model-for-one-filter&quot;&gt;Section 3&lt;/a&gt;. For comparing orders, we assume full forward passes and use \(K=n/C\).&lt;/p&gt;

&lt;p&gt;The &lt;em&gt;scan cost&lt;/em&gt; is the first filter’s latency per document. The &lt;em&gt;ask cost&lt;/em&gt; is a later filter’s latency per surviving document. Using the counts above,&lt;/p&gt;

\[\begin{aligned}
\text{scan}_{\pi_1}
&amp;amp;= \frac{T_{\text{filter}}(n_1,A_1,W_1,R_1)}{N}, \\[0.8em]
\text{ask}_{\pi_j}
&amp;amp;= \frac{T_{\text{filter}}(n_j,A_j,W_j,R_j)}{N_j},
\qquad j&amp;gt;1,\ N_j&amp;gt;0.
\end{aligned}\]

&lt;p&gt;If no documents reach a filter, it adds no cost.&lt;/p&gt;

&lt;p&gt;The first filter incurs its scan cost for all \(N\) documents. Each later filter incurs its ask cost for the \(N_j\) documents that reach it. We add these costs to obtain an &lt;em&gt;ordering score&lt;/em&gt;, \(S(\pi)\):&lt;/p&gt;

\[S(\pi) = N\,\text{scan}_{\pi_1}
       + \sum_{j=2}^{m} N_j\,\text{ask}_{\pi_j}.\]

&lt;h3 id=&quot;43-choosing-the-filter-order&quot;&gt;4.3 Choosing the Filter Order&lt;/h3&gt;

&lt;p&gt;We now choose the order with the lowest ordering score. Our ordering rule follows Joseph Hellerstein and Michael Stonebraker’s work on ordering expensive predicates.&lt;sup&gt;&lt;a href=&quot;#note-19&quot;&gt;19&lt;/a&gt;&lt;/sup&gt; For filters after the first position, we rank each filter by its ask cost per document rejected:&lt;/p&gt;

\[\text{rank}_i = \frac{\text{ask}_i}{1-s_i}.\]

&lt;p&gt;Here, \(\text{rank}_i\) divides the filter’s ask cost by the fraction of documents it rejects. Lower ranks favor filters that reject more documents for less cost. If a filter passes every document (\(s_i=1\)), we set \(\text{rank}_i=\infty\).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Algorithm 1. Filter ordering (pseudocode).&lt;/strong&gt;&lt;/p&gt;

&lt;div class=&quot;algorithm-pseudocode&quot;&gt;
  &lt;div&gt;\(\sigma \leftarrow\) filters sorted by ascending rank&lt;/div&gt;
  &lt;div&gt;for each filter \(F_i\):&lt;/div&gt;
  &lt;div class=&quot;algorithm-indent&quot;&gt;\(\pi^{(i)} \leftarrow\) \(F_i\), followed by the filters in \(\sigma\) except \(F_i\)&lt;/div&gt;
  &lt;div class=&quot;algorithm-indent&quot;&gt;evaluate \(S(\pi^{(i)})\) using the equation in &lt;a href=&quot;#42-cost-of-a-fixed-filter-order&quot;&gt;Section 4.2&lt;/a&gt;&lt;/div&gt;
  &lt;div&gt;return the candidate \(\pi^{(i)}\) with the lowest \(S(\pi^{(i)})\)&lt;/div&gt;
&lt;/div&gt;

&lt;p&gt;Here, \(\sigma\) is the rank-sorted order, and \(\pi^{(i)}\) is the candidate order with filter \(F_i\) first. The superscript is the first filter’s number, while \(\pi_j\) denotes the filter in position \(j\). Algorithm 1 sorts only once and preserves the rank order of the remaining filters in each candidate.&lt;/p&gt;

&lt;p&gt;For example, if sorting by rank gives \(\sigma=(F_2,F_1,F_3)\), the candidates are&lt;/p&gt;

\[\begin{aligned}
\pi^{(2)}&amp;amp;=(F_2,F_1,F_3),\\
\pi^{(1)}&amp;amp;=(F_1,F_2,F_3),\\
\pi^{(3)}&amp;amp;=(F_3,F_2,F_1).
\end{aligned}\]

&lt;p&gt;The selected order, \(\pi_{\text{opt}}\), is&lt;/p&gt;

\[\pi_{\text{opt}} = \underset{\pi\,\in\,\{\pi^{(1)},\ldots,\pi^{(m)}\}}{\arg\min}\ S(\pi).\]

&lt;p&gt;We sort the filters once in \(O(m\log m)\) time. After sorting, Quail computes each candidate’s score in constant time, so choosing among all \(m\) candidates takes \(O(m\log m)\) overall.&lt;/p&gt;

&lt;details class=&quot;collapsible-section proof-card&quot;&gt;
  &lt;summary&gt;&lt;h5&gt;Proof sketch: why Algorithm 1 minimizes \(S(\pi)\)&lt;/h5&gt;&lt;/summary&gt;

  &lt;p&gt;Assume fixed, nonnegative scan and ask costs per document, and selectivities that do not change with filter order.&lt;/p&gt;

  &lt;p&gt;Fix the first filter. Let \(i\) and \(j\) be adjacent filters after the first, and let \(D\) be the expected number of documents evaluated by the earlier of the two. Their contribution to \(S(\pi)\) is \(D(\text{ask}_i+s_i\text{ask}_j)\) if \(i\) runs first, or \(D(\text{ask}_j+s_j\text{ask}_i)\) if \(j\) runs first. Either order leaves \(Ds_i s_j\) documents on average, so swapping the pair changes no other filter’s cost.&lt;/p&gt;

  &lt;p&gt;Swapping \(j,i\) to \(i,j\) changes \(S(\pi)\) by&lt;/p&gt;

\[D\left[\text{ask}_i(1-s_j)-\text{ask}_j(1-s_i)\right].\]

  &lt;p&gt;For \(s_i,s_j&amp;lt;1\), the change is nonpositive whenever&lt;/p&gt;

\[\text{rank}_i
=\frac{\text{ask}_i}{1-s_i}
\le \frac{\text{ask}_j}{1-s_j}
=\text{rank}_j.\]

  &lt;p&gt;If \(s_j=1\), the change is \(-D\,\text{ask}_j(1-s_i)\le 0\), so placing \(j\) after \(i\) cannot increase the score. If both selectivities are 1, either order has the same cost. Filters with selectivity 1 can therefore go last, as prescribed by their infinite ranks.&lt;/p&gt;

  &lt;p&gt;Sorting the remaining filters by adjacent swaps cannot increase \(S(\pi)\), so rank order is optimal for any fixed first filter. Algorithm 1 tries every first filter and selects the lowest-scoring candidate, hence minimizes \(S(\pi)\) over all orders.&lt;/p&gt;

&lt;/details&gt;

&lt;p&gt;&lt;strong&gt;SoL latency of the selected order.&lt;/strong&gt; The cost model we use for query planning, which we have described so far, is pretty simple. However, an astute reader will notice that the ordering score \(S(\pi)\) charges each filter separately. It does not allow arithmetic for one filter to overlap with memory transfers for another filter.&lt;/p&gt;

&lt;p&gt;To see why, suppose attention for the first filter requires 10 ms of arithmetic and 1 ms of memory transfers, while attention for a later filter requires 1 ms of arithmetic and 5 ms of memory transfers. Applying our cost model gives&lt;/p&gt;

\[\max(10,1) + \max(1,5) = 15~\text{ms}.\]

&lt;p&gt;But across both filters, attention requires 11 ms of arithmetic and 6 ms of memory transfers. Applying the roofline equation to the combined attention gives a lower estimate:&lt;/p&gt;

\[\max(10+1,\;1+5) = 11~\text{ms}.\]

&lt;p&gt;Achieving 11 ms would require overlapping arithmetic and memory transfers across filters, for example by pipelining documents within each batch across filters. Nevertheless, to calculate this lower bound, we first count the tokens and attention comparisons across all filters in the selected order.&lt;sup&gt;&lt;a href=&quot;#note-20&quot;&gt;20&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Let \(n\) be the total number of tokens processed and \(A\) the total number of attention comparisons. We also total the KV traffic, with \(W\) counting tokens whose new KV vectors are written and \(R\) counting tokens whose saved KV vectors are read:&lt;/p&gt;

\[\begin{aligned}
n &amp;amp;= \sum_{j=1}^{m} n_j, &amp;amp; A &amp;amp;= \sum_{j=1}^{m} A_j, \\
W &amp;amp;= \sum_{j=1}^{m} W_j, &amp;amp; R &amp;amp;= \sum_{j=1}^{m} R_j.
\end{aligned}\]

&lt;p&gt;For the query’s SoL latency estimate, we round up the forward-pass count, \(K=\lceil n/C\rceil\). Substituting the totals into the projection, attention, and MLP formulas from Section 3 gives the query’s estimated SoL latency, \(T_{\text{query}}\):&lt;/p&gt;

\[\begin{aligned}
T_{\text{query}}
&amp;amp;= \max\!\left(
\frac{2P_{\text{proj}}n}{\Pi_{fp8}},\;
\frac{b_w P_{\text{proj}}K}{\beta}
\right) \\
&amp;amp;\quad + \max\!\left(
\frac{4n_{\text{heads}}d_{\text{head}}\,\text{layers}\,A}{\Pi_{bf16}},\;
\frac{B_{\text{kv}}(W+R)}{\beta}
\right) \\
&amp;amp;\quad + \max\!\left(
\frac{2P_{\text{mlp}}n}{\Pi_{fp8}},\;
\frac{b_w P_{\text{mlp}}K}{\beta}
\right).
\end{aligned}\]

&lt;h2 id=&quot;5-example-imdb-query&quot;&gt;5. Example: IMDB Query&lt;/h2&gt;

&lt;p&gt;Here we walk through our cost model for the example query over \(N=5{,}000\) IMDB reviews&lt;sup&gt;&lt;a href=&quot;#note-21&quot;&gt;21&lt;/a&gt;&lt;/sup&gt; using Qwen3-4B on an H100. We first calculate the ordering score \(S(\pi)\) for each candidate from Algorithm 1. We then calculate the SoL latency \(T_{\text{query}}\) for the selected order, \(F_2 \to F_1 \to F_3\).&lt;/p&gt;

&lt;p&gt;The weights use FP8, while attention and KV use BF16. The cached prefix contains \(p = 2 + 298.8466 = 300.8466\) tokens per review.&lt;/p&gt;

&lt;h3 id=&quot;51-filter-ordering&quot;&gt;5.1 Filter Ordering&lt;/h3&gt;

&lt;p&gt;&lt;a href=&quot;#table-5&quot;&gt;Table 5&lt;/a&gt; lists the three filters. We assume we know each filter’s selectivity, the fraction of reviews that pass it.&lt;/p&gt;

&lt;p&gt;We calculate scan and ask costs per document using the equations from &lt;a href=&quot;#42-cost-of-a-fixed-filter-order&quot;&gt;Section 4.2&lt;/a&gt;, the H100 specs in &lt;a href=&quot;#table-1&quot;&gt;Table 1&lt;/a&gt;, and the Qwen3-4B constants in &lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt;. As in the planning model, we assume tokens are packed into full forward passes. Projections and the MLP are compute-bound at our token budget, so their latency estimates use the arithmetic times.&lt;/p&gt;

&lt;p&gt;We will calculate both costs for \(F_1\), which asks whether a review mentions a positive aspect. Its ask cost applies when it runs after another filter, and its scan cost applies when it runs first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ask cost.&lt;/strong&gt; When \(F_1\) runs after another filter, it processes only its instruction. Let \(q=q_1=51\) be the instruction’s token count. The projection arithmetic per document takes&lt;/p&gt;

\[T_{\text{proj}} = \frac{2 \times 943{,}718{,}400 \times 51}{1.979\times10^{15}} = 0.0486~\text{ms},\]

&lt;p&gt;The MLP arithmetic takes&lt;/p&gt;

\[T_{\text{mlp}} = \frac{2 \times 2{,}689{,}597{,}440 \times 51}{1.979\times10^{15}} = 0.1386~\text{ms},\]

&lt;p&gt;For attention, each of the 51 new tokens attends to the \(p\) cached tokens, itself, and earlier instruction tokens. The number of attention comparisons per document is&lt;/p&gt;

\[A = q\,p + \frac{q(q+1)}{2} = 51 \times 300.8466 + \frac{51 \times 52}{2} = 16{,}669.18.\]

&lt;p&gt;Attention reads the saved KV vectors for \(p\) tokens and writes new KV vectors for \(q\) tokens. With \(W+R=p+q=351.8466\), its latency estimate is&lt;/p&gt;

\[\begin{aligned}
T_{\text{attn}}
    &amp;amp;= \max\!\left(
        \frac{589{,}824 \times 16{,}669.18}{989.5\times10^{12}},\;
        \frac{147{,}456 \times 351.8466}{3.35\times10^{12}}\right) \\
    &amp;amp;= \max(0.0099,\ 0.0155)~\text{ms} = 0.0155~\text{ms},
\end{aligned}\]

&lt;p&gt;Memory transfers take longer than arithmetic for attention. Adding the three component latencies gives the ask cost per document:&lt;/p&gt;

\[\text{ask}_1 = T_{\text{proj}} + T_{\text{mlp}} + T_{\text{attn}} = 0.0486 + 0.1386 + 0.0155 = 0.203~\text{ms}.\]

&lt;p&gt;&lt;strong&gt;Scan cost.&lt;/strong&gt; When \(F_1\) runs first, it processes the preamble, review, and instruction. The full request contains \(r=p+q=351.8466\) tokens. The projection arithmetic per document takes&lt;/p&gt;

\[T_{\text{proj}} = \frac{2 \times 943{,}718{,}400 \times 351.8466}{1.979\times10^{15}} = 0.336~\text{ms},\]

&lt;p&gt;The MLP arithmetic takes&lt;/p&gt;

\[T_{\text{mlp}} = \frac{2 \times 2{,}689{,}597{,}440 \times 351.8466}{1.979\times10^{15}} = 0.956~\text{ms},\]

&lt;p&gt;The number of attention comparisons per document is&lt;/p&gt;

\[A = \frac{r(r+1)}{2} = \frac{351.8466 \times 352.8466}{2} = 62{,}073.94\]

&lt;p&gt;The attention arithmetic therefore takes&lt;/p&gt;

\[T_{\text{attn}} = \frac{589{,}824 \times 62{,}073.94}{989.5\times10^{12}} = 0.0370~\text{ms},\]

&lt;p&gt;We write new KV vectors for all \(r\) tokens and read no vectors saved by an earlier filter, so \(W=r\) and \(R=0\). The memory transfer time is&lt;/p&gt;

\[\frac{147{,}456 \times 351.8466}{3.35\times10^{12}} = 0.0155~\text{ms},\]

&lt;p&gt;Arithmetic takes longer than memory transfers for attention, so the scan cost per document is&lt;/p&gt;

\[\text{scan}_1 = 0.336 + 0.956 + 0.037 = 1.329~\text{ms}.\]

&lt;p&gt;We calculate the other filters’ costs the same way, using 45 instruction tokens for \(F_2\) and 49 for \(F_3\).&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-5&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Filter&lt;/th&gt;
        &lt;th&gt;Predicate&lt;/th&gt;
        &lt;th&gt;\(q_i\)&lt;/th&gt;
        &lt;th&gt;\(s_i\)&lt;/th&gt;
        &lt;th&gt;\(\text{ask}_i\) (ms)&lt;/th&gt;
        &lt;th&gt;\(\text{scan}_i\) (ms)&lt;/th&gt;
        &lt;th&gt;\(1-s_i\)&lt;/th&gt;
        &lt;th&gt;rank (ms)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_2\)&lt;/td&gt;
        &lt;td&gt;discusses the ending&lt;/td&gt;
        &lt;td&gt;45&lt;/td&gt;
        &lt;td&gt;0.2273&lt;/td&gt;
        &lt;td&gt;0.180&lt;/td&gt;
        &lt;td&gt;1.306&lt;/td&gt;
        &lt;td&gt;0.7727&lt;/td&gt;
        &lt;td&gt;0.234&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_1\)&lt;/td&gt;
        &lt;td&gt;mentions a positive aspect&lt;/td&gt;
        &lt;td&gt;51&lt;/td&gt;
        &lt;td&gt;0.4856&lt;/td&gt;
        &lt;td&gt;0.203&lt;/td&gt;
        &lt;td&gt;1.329&lt;/td&gt;
        &lt;td&gt;0.5144&lt;/td&gt;
        &lt;td&gt;0.394&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_3\)&lt;/td&gt;
        &lt;td&gt;mentions a named actor&lt;/td&gt;
        &lt;td&gt;49&lt;/td&gt;
        &lt;td&gt;0.6123&lt;/td&gt;
        &lt;td&gt;0.195&lt;/td&gt;
        &lt;td&gt;1.321&lt;/td&gt;
        &lt;td&gt;0.3877&lt;/td&gt;
        &lt;td&gt;0.504&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 5. Filters sorted by ascending rank ask&lt;sub&gt;i&lt;/sub&gt;/(1&amp;minus;s&lt;sub&gt;i&lt;/sub&gt;). Each &lt;em&gt;s&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; is the assumed fraction of reviews that pass the filter.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;Sorting by rank gives \(F_2 \to F_1 \to F_3\), rather than the written order \(F_1 \to F_2 \to F_3\). The ask costs differ by only about 13% because the instructions have similar lengths. \(F_2\) rejects the most documents per unit of estimated latency, while \(F_3\) rejects the fewest, so \(F_3\) goes last. To compare orders, we assume each filter keeps the same fraction of documents regardless of which filters ran before it.&lt;/p&gt;

&lt;p&gt;We still need to compare each possible first filter, as described in &lt;a href=&quot;#43-choosing-the-filter-order&quot;&gt;Section 4.3&lt;/a&gt;. &lt;a href=&quot;#table-6&quot;&gt;Table 6&lt;/a&gt; shows that starting with \(F_2\) gives the lowest ordering score. Starting with \(F_1\) costs \(0.324\) s, or about 4.7%, more. Its scan is more expensive, and it passes about 2,428 documents to \(F_2\). Starting with \(F_2\) passes only about 1,137 documents to \(F_1\).&lt;/p&gt;

&lt;p&gt;For the candidate \(\pi=(F_2,F_1,F_3)\), the expected numbers of reviews evaluated by each filter are&lt;/p&gt;

\[\begin{aligned}
N_1&amp;amp;=5{,}000,\\
N_2&amp;amp;=5{,}000\times s_2=1{,}136.5,\\
N_3&amp;amp;=5{,}000\times s_2\times s_1=551.8844.
\end{aligned}\]

&lt;p&gt;Substituting the counts and the scan and ask costs into \(S(\pi)\) from Section 4.2 gives&lt;/p&gt;

\[\begin{aligned}
S((F_2,F_1,F_3))
&amp;amp;=5{,}000\,\text{scan}_2
 +1{,}136.5\,\text{ask}_1
 +551.8844\,\text{ask}_3\\
&amp;amp;\approx 6.8665~\text{s}.
\end{aligned}\]

&lt;p&gt;We use unrounded scan and ask costs when calculating the scores. Table 6 compares the candidates from Algorithm 1.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-6&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;First&lt;/th&gt;
        &lt;th&gt;Ordering \(\pi\)&lt;/th&gt;
        &lt;th&gt;\(S(\pi)\) (s)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_2\)&lt;/td&gt;
        &lt;td&gt;\(F_2 \to F_1 \to F_3\)&lt;/td&gt;
        &lt;td&gt;6.8665&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_1\)&lt;/td&gt;
        &lt;td&gt;\(F_1 \to F_2 \to F_3\)&lt;/td&gt;
        &lt;td&gt;7.1906&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;\(F_3\)&lt;/td&gt;
        &lt;td&gt;\(F_3 \to F_2 \to F_1\)&lt;/td&gt;
        &lt;td&gt;7.2994&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 6. Ordering score for each candidate ordering.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;We select \(F_2 \to F_1 \to F_3\), which has the lowest ordering score, \(S(\pi)=6.8665\) s.&lt;/p&gt;

&lt;h3 id=&quot;52-sol-estimate-for-the-ordered-conjunction&quot;&gt;5.2 SoL Estimate for the Ordered Conjunction&lt;/h3&gt;

&lt;p&gt;We now calculate \(T_{\text{query}}\) for the same order, \(\pi=(F_2,F_1,F_3)\), using the query-wide roofline equation from &lt;a href=&quot;#43-choosing-the-filter-order&quot;&gt;Section 4.3&lt;/a&gt;. Unlike \(S(\pi)\), we first total the token, attention, and KV counts across filters rather than adding the filters’ latency estimates.&lt;/p&gt;

&lt;p&gt;The capacity calculation in &lt;a href=&quot;#41-kv-reuse-and-hbm-capacity&quot;&gt;Section 4.1&lt;/a&gt; shows that only \(1{,}204\) document prefixes fit in HBM at once, so the \(5{,}000\) documents run as five batches.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#table-7&quot;&gt;Table 7&lt;/a&gt; applies the first-filter and later-filter formulas from &lt;a href=&quot;#42-cost-of-a-fixed-filter-order&quot;&gt;Section 4.2&lt;/a&gt;, using the expected document counts 5,000, 1,136.5, and 551.8844 without rounding. We also keep the resulting token, attention, and KV counts unrounded in the calculations. The table rounds those work counts only for display.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-7&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Stage&lt;/th&gt;
        &lt;th&gt;\(n_j\)&lt;/th&gt;
        &lt;th&gt;\(A_j\)&lt;/th&gt;
        &lt;th&gt;\(W_j\)&lt;/th&gt;
        &lt;th&gt;\(R_j\)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;1 (\(F_2\))&lt;/td&gt;
        &lt;td&gt;1,729,233&lt;/td&gt;
        &lt;td&gt;\(2.999\times10^{8}\)&lt;/td&gt;
        &lt;td&gt;1,729,233&lt;/td&gt;
        &lt;td&gt;0&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;2 (\(F_1\))&lt;/td&gt;
        &lt;td&gt;57,962&lt;/td&gt;
        &lt;td&gt;\(1.894\times10^{7}\)&lt;/td&gt;
        &lt;td&gt;57,962&lt;/td&gt;
        &lt;td&gt;341,912&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;3 (\(F_3\))&lt;/td&gt;
        &lt;td&gt;27,042&lt;/td&gt;
        &lt;td&gt;\(8.812\times10^{6}\)&lt;/td&gt;
        &lt;td&gt;27,042&lt;/td&gt;
        &lt;td&gt;166,033&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;strong&gt;Conjunction&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;1,814,237&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;\(3.276\times10^{8}\)&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;1,814,237&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;507,945&lt;/strong&gt;&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 7. Per-stage work of the ordered IMDB conjunction, and its totals.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;The model’s forward-pass count assumes tokens can be packed across filters:&lt;/p&gt;

\[K=\left\lceil\frac{n}{C}\right\rceil
 =\left\lceil\frac{1{,}814{,}236.8356}{110{,}376}\right\rceil
 =17.\]

&lt;p&gt;Attention’s HBM traffic is \(B_{\text{attn}}=B_{\text{kv}}(W+R)\), with \(W+R\approx2{,}322{,}182\) tokens. Table 8 lists the resulting arithmetic and memory times for each component.&lt;/p&gt;

&lt;p&gt;Attention explains the difference between the ordering score and the query-wide estimate. It is compute-bound for the first filter but memory-bound for the later filters. Adding the separate attention estimates gives&lt;/p&gt;

\[178.76+17.60+8.50\approx204.86~\text{ms}.\]

&lt;p&gt;The query-wide estimate instead uses the total arithmetic and memory times:&lt;/p&gt;

\[\max(195.30,\;102.22)=195.30~\text{ms}.\]

&lt;p&gt;The approximately 9.55 ms difference assumes memory transfers can overlap with arithmetic across filters, as discussed in Section 4.3. Projections and the MLP are compute-bound in every stage, so their latency estimates agree under both calculations.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-8&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th&gt;Component&lt;/th&gt;
        &lt;th&gt;FLOPs&lt;/th&gt;
        &lt;th&gt;Bytes&lt;/th&gt;
        &lt;th&gt;\(T_{\text{compute}}\)&lt;/th&gt;
        &lt;th&gt;\(T_{\text{memory}}\)&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td&gt;Projections&lt;/td&gt;
        &lt;td&gt;\(3.42\times10^{15}\)&lt;/td&gt;
        &lt;td&gt;\(1.60\times10^{10}\)&lt;/td&gt;
        &lt;td&gt;1.7303 s&lt;/td&gt;
        &lt;td&gt;0.0048 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;Attention&lt;/td&gt;
        &lt;td&gt;\(1.93\times10^{14}\)&lt;/td&gt;
        &lt;td&gt;\(3.42\times10^{11}\)&lt;/td&gt;
        &lt;td&gt;0.1953 s&lt;/td&gt;
        &lt;td&gt;0.1022 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;MLP&lt;/td&gt;
        &lt;td&gt;\(9.76\times10^{15}\)&lt;/td&gt;
        &lt;td&gt;\(4.57\times10^{10}\)&lt;/td&gt;
        &lt;td&gt;4.9313 s&lt;/td&gt;
        &lt;td&gt;0.0136 s&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;\(1.34\times10^{16}\)&lt;/td&gt;
        &lt;td&gt;\(4.04\times10^{11}\)&lt;/td&gt;
        &lt;td&gt;&lt;strong&gt;6.857 s&lt;/strong&gt;&lt;/td&gt;
        &lt;td&gt;0.12 s&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 8. Query-wide roofline estimate for the ordered IMDB conjunction. For each component&apos;s total work, arithmetic takes longer than memory transfers.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;Adding the component latencies from &lt;a href=&quot;#table-8&quot;&gt;Table 8&lt;/a&gt; gives the SoL estimate for \(F_2 \to F_1 \to F_3\):&lt;/p&gt;

\[\begin{aligned}
T_{\text{query}}
    &amp;amp;= \max(1.7303,\;0.0048) \\
    &amp;amp;\quad + \max(0.1953,\;0.1022) \\
    &amp;amp;\quad + \max(4.9313,\;0.0136) \\
    &amp;amp;= 1.7303 + 0.1953 + 4.9313 \\
    &amp;amp;= 6.857~\text{seconds} \\
    &amp;amp;\approx \boxed{6.86~\text{seconds}} .
\end{aligned}\]

&lt;p&gt;For \(\pi=(F_2,F_1,F_3)\), we used \(S(\pi)=6.8665\) s to choose the order and obtained \(T_{\text{query}}\approx6.857\) s as its SoL estimate. The roughly 9.6 ms difference comes from estimating attention for each filter separately in \(S(\pi)\) rather than assuming overlap across filters in \(T_{\text{query}}\).&lt;/p&gt;

&lt;p&gt;Phew! We made it! The SoL estimate is about 6.86 seconds for all 5,000 reviews. Of course, actual runtime may be much longer. Sustaining NVIDIA’s peak arithmetic throughput and memory bandwidth, with enough overlap across filters, may not be achievable for this query. But the estimate gives us a lower bound to compare against as we improve execution.&lt;/p&gt;

&lt;h2 id=&quot;6-filter-playground&quot;&gt;6. Filter Playground&lt;/h2&gt;

&lt;p&gt;To help illustrate how filter ordering affects query cost, we’ve built an interactive playground! You can change the filter order, selectivities, and instruction lengths to see how they affect the query’s SoL estimate. The playground uses Qwen3-4B-fp8 on an H100, starting with the IMDB example from &lt;a href=&quot;#5-example-imdb-query&quot;&gt;Section 5&lt;/a&gt;. Drag filters to reorder them, or apply the ordering rule from &lt;a href=&quot;#43-choosing-the-filter-order&quot;&gt;Section 4.3&lt;/a&gt;. For up to six filters, you can compare the rule’s order with every possible order. You can also switch to a hybrid model, Qwen3.5-4B-fp8 or Qwen3.5-27B-fp8, where three of every four layers use Gated DeltaNet linear attention and keep a fixed-size state per document instead of KV; the playground prices them as Quail does.&lt;/p&gt;

&lt;p&gt;The estimates assume peak hardware rates and reuse of each document’s cached prefix across filters. We use the average document length for every document.&lt;/p&gt;

&lt;div class=&quot;pg&quot; id=&quot;filter-chain-playground&quot; data-filter-chain-calculator=&quot;&quot;&gt;
  &lt;noscript&gt;The playground needs JavaScript. It starts with the three IMDB filters from Section 5, where the ordering rule picks F2 → F1 → F3 with an estimated 6.87 s, compared with 7.30 s for the slowest candidate order (Table 6).&lt;/noscript&gt;
&lt;/div&gt;
&lt;script src=&quot;https://cdnjs.cloudflare.com/ajax/libs/Sortable/1.15.6/Sortable.min.js&quot; defer=&quot;&quot;&gt;&lt;/script&gt;

&lt;script src=&quot;/assets/js/filter-chain-calculator.js&quot; defer=&quot;&quot;&gt;&lt;/script&gt;

&lt;h2 id=&quot;7-conclusion&quot;&gt;7. Conclusion&lt;/h2&gt;

&lt;p&gt;In this article, we walked through costing one AI-powered filter and then a conjunction of filters. Wow, there were a lot of steps involved! We had to account for arithmetic, memory traffic, KV reuse, and filter ordering. Even for a filter that returns just true or false, there’s a lot to account for!&lt;/p&gt;

&lt;p&gt;And this is just the tip of the iceberg. We need to extend these calculations to other operators, such as AI JOIN and AI CLASSIFY, and to different models and hardware. Hybrid models such as Qwen3.5 mix linear and full attention, so we need to account for different arithmetic and saved state.&lt;sup&gt;&lt;a href=&quot;#note-22&quot;&gt;22&lt;/a&gt;&lt;/sup&gt; Then on the hardware side, for example, NVIDIA’s Blackwell GPUs add native support for four-bit floating-point (FP4) arithmetic, allowing smaller weights and higher peak throughput than FP8.&lt;sup&gt;&lt;a href=&quot;#note-23&quot;&gt;23&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Overall, we think there’s a lot of cool work in this space for the database community to take on! Model and hardware choices now affect how we cost operators, choose query plans, and schedule execution. We hope this article gives you a useful starting point if you’re building an AI-powered query engine or trying to understand how much faster your system could be.&lt;/p&gt;

&lt;h2 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h2&gt;

&lt;p&gt;We thank &lt;a href=&quot;https://modal.com/&quot;&gt;Modal&lt;/a&gt; for sponsoring the compute used in
this research.&lt;/p&gt;

&lt;h2 id=&quot;notes&quot;&gt;Notes&lt;/h2&gt;

&lt;p&gt;&lt;span id=&quot;note-1&quot;&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/span&gt; Shreya Shankar, Charles Frye, Fergus Finn, Arnav Dhariya, Joseph Barrow, and Meryem Arik. 2026. Building an Ultra-High Throughput AI-SQL Engine. &lt;a href=&quot;https://fsdatalab.github.io/blog/introducing-quail/&quot;&gt;https://fsdatalab.github.io/blog/introducing-quail/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-2&quot;&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/span&gt; Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. &lt;em&gt;Commun. ACM&lt;/em&gt; 52, 4, 65–76. &lt;a href=&quot;https://doi.org/10.1145/1498765.1498785&quot;&gt;doi:10.1145/1498765.1498785&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-3&quot;&gt;&lt;strong&gt;3.&lt;/strong&gt;&lt;/span&gt; The H100 SXM memory capacity, bandwidth, and Tensor Core throughput come from &lt;a href=&quot;https://resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306&quot;&gt;NVIDIA’s H100 specifications&lt;/a&gt;. NVIDIA reports Tensor Core throughput for structured sparse matrices. Table 1 uses dense throughput, which is half the reported sparse throughput. The L2 cache size, SM count, and number of Tensor Cores per SM come from &lt;a href=&quot;https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/&quot;&gt;NVIDIA’s Hopper architecture overview&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-4&quot;&gt;&lt;strong&gt;4.&lt;/strong&gt;&lt;/span&gt; Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. &lt;a href=&quot;https://arxiv.org/abs/2205.14135&quot;&gt;FlashAttention: fast and memory-efficient exact attention with IO-awareness&lt;/a&gt;. In &lt;em&gt;Advances in Neural Information Processing Systems 35 (NeurIPS 2022)&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-5&quot;&gt;&lt;strong&gt;5.&lt;/strong&gt;&lt;/span&gt; An Yang et al. 2025. Qwen3 Technical Report. &lt;a href=&quot;https://arxiv.org/abs/2505.09388&quot;&gt;arXiv:2505.09388&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-6&quot;&gt;&lt;strong&gt;6.&lt;/strong&gt;&lt;/span&gt; We use Qwen3-4B FP8 because its grouped query attention reduces KV-cache storage and its FP8 weights use the H100’s higher FP8 throughput. Its 32 query heads share eight key-value heads, making the KV cache one quarter the size of full multi-head attention. FP8 also halves raw weight storage relative to BF16, while the KV cache remains in BF16. See &lt;a href=&quot;https://docs.nvidia.com/deeplearning/transformer-engine-releases/release-2.5/user-guide/examples/fp8_primer.html&quot;&gt;NVIDIA’s FP8 primer&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-7&quot;&gt;&lt;strong&gt;7.&lt;/strong&gt;&lt;/span&gt; Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. &lt;a href=&quot;https://aclanthology.org/2023.emnlp-main.298/&quot;&gt;GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints&lt;/a&gt;. In &lt;em&gt;Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing&lt;/em&gt;, 4895–4901.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-8&quot;&gt;&lt;strong&gt;8.&lt;/strong&gt;&lt;/span&gt; &lt;a href=&quot;https://arxiv.org/abs/2307.09288&quot;&gt;Llama 2&lt;/a&gt; 7B and 13B use multi-head attention, while Llama 2 70B uses GQA. Multi-head attention stores a separate key and value head for every query head, which increases the KV-cache footprint. For workloads that retain many document prefixes, we recommend that the database community prioritize models with smaller cache footprints, such as GQA models or &lt;a href=&quot;https://qwen.ai/blog?id=qwen3-next&quot;&gt;hybrid models&lt;/a&gt; that use full attention in only some layers.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-9&quot;&gt;&lt;strong&gt;9.&lt;/strong&gt;&lt;/span&gt; Noam Shazeer. 2020. GLU Variants Improve Transformer. &lt;a href=&quot;https://arxiv.org/abs/2002.05202&quot;&gt;arXiv:2002.05202&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-10&quot;&gt;&lt;strong&gt;10.&lt;/strong&gt;&lt;/span&gt; Kipply. 2022. Transformer Inference Arithmetic. &lt;a href=&quot;https://kipp.ly/transformer-inference-arithmetic/&quot;&gt;https://kipp.ly/transformer-inference-arithmetic/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-11&quot;&gt;&lt;strong&gt;11.&lt;/strong&gt;&lt;/span&gt; Fergus Finn. 2026. The economics of speculative decoding. Blog post. &lt;a href=&quot;https://fergusfinn.com/blog/economics-of-speculative-decoding/&quot;&gt;https://fergusfinn.com/blog/economics-of-speculative-decoding/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-12&quot;&gt;&lt;strong&gt;12.&lt;/strong&gt;&lt;/span&gt; Ben Mayer. 2026. HTDYM (How To Deploy Your Model). Sail Research blog. &lt;a href=&quot;https://www.sailresearch.com/blog/htdym&quot;&gt;https://www.sailresearch.com/blog/htdym&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-13&quot;&gt;&lt;strong&gt;13.&lt;/strong&gt;&lt;/span&gt; Modal. LLM Engineer’s Almanac (Spec Dec Roofline Model / Speedup ratio). Web page. &lt;a href=&quot;https://modal.com/llm-almanac/spec-dec-roofline&quot;&gt;https://modal.com/llm-almanac/spec-dec-roofline&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-14&quot;&gt;&lt;strong&gt;14.&lt;/strong&gt;&lt;/span&gt; We simplify attention memory traffic by counting new KV writes and reads of previously cached KV vectors. We omit reads of newly computed KV vectors and repeated reads within attention.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-15&quot;&gt;&lt;strong&gt;15.&lt;/strong&gt;&lt;/span&gt; NVIDIA’s matrix multiplication guide counts each fused multiply–add as two operations, so a product of \(M\times K\) and \(K \times N\) matrices takes \(2MKN\) FLOPs. See &lt;a href=&quot;https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#math-mem&quot;&gt;NVIDIA’s Matrix Multiplication Background&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-16&quot;&gt;&lt;strong&gt;16.&lt;/strong&gt;&lt;/span&gt; Quail reserves two \(C\)-token activation buffers, budgeting \(32d_{\text{model}}\) bytes per token, for about 18.08 GB in our example. The factor 32 is an implementation estimate. For comparison, &lt;a href=&quot;https://github.com/EleutherAI/cookbook/blob/main/calc/calc_transformer_mem.py&quot;&gt;EleutherAI’s memory calculator&lt;/a&gt; uses \(34d_{\text{model}}\) bytes per token for its 16-bit inference activation estimate.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-17&quot;&gt;&lt;strong&gt;17.&lt;/strong&gt;&lt;/span&gt; We assume each filter has the same selectivity regardless of which filters ran before it. Multiplying the selectivities estimates the fraction of documents that pass every preceding filter. Correlated filter outcomes can make that estimate inaccurate.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-18&quot;&gt;&lt;strong&gt;18.&lt;/strong&gt;&lt;/span&gt; You might be tempted to think attention is always compute-bound in a large batch, but a short filter question can make it memory-bound. For example, \(F_1\) in our IMDB query asks whether a review mentions a positive aspect. When it runs after another filter, attention reads the saved KV vectors for a roughly 301-token prefix (the 2-token shared preamble plus the average review’s 299 tokens) and writes new KV vectors for its 51-token instruction. With Qwen3-4B on an H100, attention’s arithmetic takes about 0.010 ms per document, but the KV reads and writes take 0.015 ms. Thus, the memory transfers take longer, even when we process many documents together.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-19&quot;&gt;&lt;strong&gt;19.&lt;/strong&gt;&lt;/span&gt; Joseph M. Hellerstein and Michael Stonebraker. 1993. Predicate Migration: Optimizing Queries with Expensive Predicates. In &lt;em&gt;Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data&lt;/em&gt;, 267–276. &lt;a href=&quot;https://doi.org/10.1145/170035.170078&quot;&gt;doi:10.1145/170035.170078&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-20&quot;&gt;&lt;strong&gt;20.&lt;/strong&gt;&lt;/span&gt; In our cost model, we give each filter a fixed scan cost and ask cost per document. We chose this simpler model so we could apply Hellerstein and Stonebraker’s expensive-predicate ordering rule and choose an order in \(O(m\log m)\) time for \(m\) filters. When every component of every filter is compute-bound, the rule is optimal for the SoL estimate too (no formal proof here, but you can intuit this is true). However, filters with different arithmetic and memory bottlenecks may need a different ordering rule. Finding an efficient rule for that case is still TBD. Surely, if GPT Astra can solve Navier-Stokes, it can tell us whether we can find the order with the lowest SoL estimate in \(O(m\log m)\) time too?&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-21&quot;&gt;&lt;strong&gt;21.&lt;/strong&gt;&lt;/span&gt; Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In &lt;em&gt;Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies&lt;/em&gt;, 142–150. &lt;a href=&quot;https://aclanthology.org/P11-1015/&quot;&gt;https://aclanthology.org/P11-1015/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-22&quot;&gt;&lt;strong&gt;22.&lt;/strong&gt;&lt;/span&gt; Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. &lt;a href=&quot;https://qwen.ai/blog?id=qwen3.5&quot;&gt;Qwen3.5 announcement&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-23&quot;&gt;&lt;strong&gt;23.&lt;/strong&gt;&lt;/span&gt; See &lt;a href=&quot;https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/&quot;&gt;NVIDIA’s Blackwell architecture overview&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&quot;cite-this-article&quot;&gt;Cite this article&lt;/h2&gt;

&lt;div class=&quot;bibtex-block&quot;&gt;
  &lt;p&gt;&lt;button class=&quot;copy-bibtex&quot; type=&quot;button&quot;&gt;Copy BibTeX&lt;/button&gt;&lt;/p&gt;

  &lt;div class=&quot;language-bibtex highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nc&quot;&gt;@misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;dhariya2026aifilter&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;title&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{How to Cost Your AI-Powered Filters}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;author&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Dhariya, Arnav A. and Shankar, Shreya}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;year&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{2026}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;month&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;sep&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;url&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{https://fsdatalab.github.io/blog/ai-filter-cost-estimates/}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

&lt;/div&gt;
</content>
  </entry>
  <entry>
    <title>Building an Ultra-High Throughput AI-SQL Engine</title>
    <link href="https://fsdatalab.github.io/blog/introducing-quail/" rel="alternate" type="text/html"/>
    <id>https://fsdatalab.github.io/blog/introducing-quail/</id>
    <published>2026-09-24T00:00:00+00:00</published>
    <updated>2026-09-24T00:00:00+00:00</updated>
    <author><name>Shreya Shankar</name><uri>https://www.sh-reya.com/</uri></author>
    <author><name>Charles Frye</name></author>
    <author><name>Fergus Finn</name></author>
    <author><name>Arnav Dhariya</name><uri>https://arnavdhariya.github.io/</uri></author>
    <author><name>Joseph Barrow</name></author>
    <author><name>Meryem Arik</name></author>
    <summary>Quail jointly plans AI-SQL queries and model inference. Across 29 QUAIL-B queries, it is 1.84x faster on average than well-tuned vLLM baselines.</summary>
    <content type="html">&lt;aside class=&quot;tldr&quot;&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt;, the &lt;strong&gt;QU&lt;/strong&gt;ery-&lt;strong&gt;A&lt;/strong&gt;ware &lt;strong&gt;I&lt;/strong&gt;nference &lt;strong&gt;L&lt;/strong&gt;ayer, is an open-source inference engine for AI functions in SQL. It jointly optimizes query planning and model inference for open-weight models, instead of relying on costly, closed LLM APIs. Across 29 &lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt; queries, Quail is 1.84x faster on average than well-tuned vLLM baselines, and up to 14x faster. &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Star us on GitHub&lt;/a&gt;, and try it out in our &lt;a href=&quot;https://fsdatalab--quail-playground-page.modal.run/&quot;&gt;live demo&lt;/a&gt;!&lt;/aside&gt;

&lt;nav class=&quot;post-toc&quot; aria-label=&quot;Table of contents&quot;&gt;
&lt;strong&gt;Contents&lt;/strong&gt;
&lt;ol&gt;
  &lt;li&gt;&lt;a href=&quot;#1-ai-sql-makes-unstructured-data-useful-but-it-is-expensive&quot;&gt;AI-SQL makes unstructured data useful, but it is expensive.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#2-key-idea-query-plans-should-control-llm-inference&quot;&gt;Key Idea: Query plans should control LLM inference!&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#3-we-built-quail-to-run-ai-sql-queries-faster&quot;&gt;We built Quail to run AI-SQL queries faster.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#31-you-can-run-your-first-quail-query-in-a-few-lines-of-code&quot;&gt;Run your first Quail query.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#32-quail-jointly-plans-queries-and-inference&quot;&gt;How Quail plans queries and inference.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#4-we-evaluate-quail-against-vllm&quot;&gt;We evaluate Quail against vLLM.&lt;/a&gt;
    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#41-metrics-and-baselines-for-ai-sql-performance&quot;&gt;Metrics and baselines for AI-SQL performance.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#42-overall-quail-is-184x-faster-across-quail-b&quot;&gt;Overall, Quail is 1.84x faster across QUAIL-B.&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#43-quail-dominates-vllm-on-bio-4-1404x-faster&quot;&gt;Quail dominates vLLM on BIO-4: 14.04x faster!&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#44-but-vllm-dominates-quail-on-agent-1-quail-takes-232x-as-long&quot;&gt;But, vLLM dominates Quail on AGENT-1: Quail takes 2.32x as long.&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#5-put-another-way-quail-brings-jev-like-speeds-and-intelligence-to-database-scale-workloads&quot;&gt;Put another way: Quail brings Jev-like speeds and intelligence to database-scale workloads.&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#6-we-are-just-getting-started-with-quail&quot;&gt;We are just getting started with Quail!&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/nav&gt;

&lt;h2 id=&quot;1-ai-sql-makes-unstructured-data-useful-but-it-is-expensive&quot;&gt;1. AI-SQL makes unstructured data useful, but it is expensive.&lt;/h2&gt;

&lt;p&gt;Fast LLM classifiers have been taking over the internet lately.
&lt;a href=&quot;https://typesafe.ai/blog/introducing-system-one-models-and-jev&quot;&gt;Jev&lt;/a&gt; is
the clearest example: give a model a small, bounded decision and get an
answer almost immediately. What better place to run millions of those
decisions than… inside the database!&lt;/p&gt;

&lt;p&gt;Indeed, database vendors have recently begun to offer this kind of
intelligence at scale through AI-SQL, also called AI functions. AI-SQL
extends SQL with user-defined functions that invoke LLMs. Users specify
each function with a natural-language prompt. A query can look like this:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reviews&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;WHERE&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;Does this review discuss the ending?&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n\n&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;{0}&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;review&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Many database vendors support AI-SQL. For example, &lt;a href=&quot;https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql&quot;&gt;Snowflake Cortex
AISQL&lt;/a&gt;,
&lt;a href=&quot;https://cloud.google.com/bigquery/docs/generative-ai-overview&quot;&gt;BigQuery AI
functions&lt;/a&gt;,
&lt;a href=&quot;https://docs.databricks.com/aws/en/large-language-models/ai-functions&quot;&gt;Databricks AI
Functions&lt;/a&gt;,
and, recently, &lt;a href=&quot;https://motherduck.com/blog/motherduck-supports-jev/&quot;&gt;MotherDuck&lt;/a&gt;
all support it.&lt;/p&gt;

&lt;p&gt;Unfortunately, executing AI-SQL is extremely expensive. An AI function
evaluates its prompt row by row, so one SQL query can create hundreds of
thousands or millions of model calls. A filter needs one LLM call per row. A
naive join needs one LLM call for every pair of rows in its two input tables.&lt;/p&gt;

&lt;p&gt;This line of work has become extremely popular in the database research
community. A number of open-source academic systems have emerged, including our work on
&lt;a href=&quot;https://docetl.org/&quot;&gt;DocETL&lt;/a&gt; from UC Berkeley,
&lt;a href=&quot;https://lotus-data.github.io/&quot;&gt;LOTUS&lt;/a&gt; from Stanford,
&lt;a href=&quot;https://palimpzest.org/&quot;&gt;Palimpzest&lt;/a&gt; from MIT, and
&lt;a href=&quot;https://github.com/itrummer/thalamusdb&quot;&gt;ThalamusDB&lt;/a&gt; from Cornell. These systems
(and database vendors) primarily reduce cost by eliminating as many LLM calls as possible
(e.g., &lt;a href=&quot;https://arxiv.org/abs/2512.02289&quot;&gt;MOAR&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/2601.05536&quot;&gt;Task
Cascades&lt;/a&gt;, and
&lt;a href=&quot;https://arxiv.org/abs/2505.14661&quot;&gt;Abacus&lt;/a&gt;) and by using cheaper models when
possible (e.g., &lt;a href=&quot;https://arxiv.org/abs/2509.02896&quot;&gt;BARGAIN&lt;/a&gt;). Even after these
optimizations, a query plan may still require hundreds of thousands or millions
of LLM calls.&lt;/p&gt;

&lt;h2 id=&quot;2-key-idea-query-plans-should-control-llm-inference&quot;&gt;2. Key Idea: Query plans should control LLM inference!&lt;/h2&gt;

&lt;p&gt;A natural thought is to use a general-purpose inference engine such as
vLLM to execute the query plan. However, sending millions of related
model calls to vLLM as separate requests has a large cost! We’ll
illustrate with the following query:&lt;/p&gt;

&lt;p class=&quot;query-statement&quot;&gt;Given a dataset of medical reports and a dataset of
possible adverse reactions, find serious adverse event reports that mention
both a cardiovascular reaction and a neurological reaction.&lt;sup&gt;&lt;a href=&quot;#note-1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;We call this query BIO-4 in
&lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt;, a benchmark we are
building to evaluate AI-SQL query engines. Its inputs contain 5,000 long
reports and 4,144 reaction terms (the latter is used twice, as there are
two joins). The logical plan, shown in &lt;a href=&quot;#figure-1&quot;&gt;Figure 1&lt;/a&gt;, works as follows:&lt;/p&gt;

&lt;figure class=&quot;figure-medium&quot; id=&quot;figure-1&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/bio4-logical-plan.svg&quot; alt=&quot;The BIO-4 logical query plan.&quot; /&gt;
  &lt;figcaption&gt;Figure 1. A logical query plan for the BIO-4 query plan filters all three inputs before the joins. Both joins use the medical report as the anchor (i.e., first document in the prompt).&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;It filters the reports for serious adverse events.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It filters the two reaction term dataset inputs (i.e., lists of
possible adverse reaction terms) for cardiovascular and neurological
reactions.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It joins the surviving reports with the cardiovascular terms, then
with the neurological terms.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;How might we execute BIO-4 with vLLM? Following what databases do, we’d
render one prompt for each filter input, and, for the join, one prompt
for candidate report and reaction &lt;em&gt;pair&lt;/em&gt;. Each prompt would be a
separate inference request.&lt;sup&gt;&lt;a href=&quot;#note-2&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; For the filters, there’s just one document
per prompt; for the join, we’d place the much longer medical report
first as the anchor and the reaction term second as the partner to
maximize reuse of the prefix’s key and value state (KV) across join
prompts. We’d execute one operator at a time and order its LLM requests
so requests for the same document are evaluated together, maximizing
KV reuse.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A cost estimate for the query plan.&lt;/strong&gt; Before running the vLLM
baseline, we first want to estimate the lowest possible runtime for the
same plan. We count the model’s arithmetic work and HBM traffic from the
token lengths, then use a &lt;a href=&quot;https://modal.com/gpu-glossary/perf/roofline-model&quot;&gt;roofline
model&lt;/a&gt; to estimate
the time. The estimate assumes peak GPU throughput, full overlap between
CPU and GPU work, and unlimited space for retained KV. No implementation
can meet all of these assumptions, so this optimistic lower bound is our
&lt;em&gt;speed of light estimate&lt;/em&gt;, or SoL. For BIO-4, the SoL estimate is 894.37
seconds, or 14.91 minutes.&lt;sup&gt;&lt;a href=&quot;#note-3&quot;&gt;3&lt;/a&gt;&lt;/sup&gt; The &lt;a href=&quot;https://github.com/fsdatalab/quail-exploration/blob/0d24478a82100b518d6110f5c1c8cec0c26c6487/quail/planner/sol.py&quot;&gt;implementation in
Quail&lt;/a&gt;
contains the full calculation (which we’ll discuss in a follow-up blog
post).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How we hoped vLLM would perform.&lt;/strong&gt; Each request produces one token
constrained to TRUE or FALSE, so all model work is prefill. BIO-4
compiles to millions of requests, so there should always be a large
batch ready for the H100. With a large enough batch, vLLM should keep
the H100 busy, and get as close to the SoL estimate as possible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How vLLM actually performs.&lt;/strong&gt; We run vLLM 0.26.0 with Qwen3 4B FP8 on
one H100. We give it enough batch capacity to use the GPU. At scale
factor 1.0, a vLLM baseline takes &lt;em&gt;6.84 hours&lt;/em&gt;, or 27.55x the SoL
estimate! There are two reasons for the inefficiency. &lt;a href=&quot;#figure-2&quot;&gt;Figure 2&lt;/a&gt; shows a
representative 3.5-second window from the first join.&lt;/p&gt;

&lt;figure class=&quot;figure-profile&quot; id=&quot;figure-2&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/vllm-bio4-perfetto.jpg&quot; alt=&quot;A PyTorch profiler trace of the first BIO-4 join in the vLLM baseline.&quot; /&gt;
  &lt;figcaption&gt;Figure 2. A 3.5-second window from the first BIO-4 join from a PyTorch profiler trace. The CPU main thread alternates between vLLM scheduler work and execute_context. GPU stream 25 runs during the three execute_context blocks; as you can see, the gaps between them leave the H100 idle.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;The primary reason is host overhead. The CPU spends long stretches
scheduling and tracking requests while the H100 waits, as shown by the
gaps in &lt;a href=&quot;#figure-2&quot;&gt;Figure 2&lt;/a&gt;.&lt;sup&gt;&lt;a href=&quot;#note-4&quot;&gt;4&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The second is &lt;em&gt;KV regret:&lt;/em&gt; vLLM sometimes discards KV that it needs
again later. On BIO-4 at scale factor 1.0, it therefore processes 174.6
million tokens instead of 124.3 million: 50.3 million extra tokens, or
40% more work.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We can, and we should, reduce both sources of waste by optimizing
inference for AI-SQL!&lt;/p&gt;

&lt;h2 id=&quot;3-we-built-quail-to-run-ai-sql-queries-faster&quot;&gt;3. We built Quail to run AI-SQL queries faster.&lt;/h2&gt;

&lt;p&gt;We will first show you how you can get started. To skip to read about
how Quail works, skip to &lt;a href=&quot;#32-quail-jointly-plans-queries-and-inference&quot;&gt;Section 3.2&lt;/a&gt;.&lt;/p&gt;

&lt;details class=&quot;collapsible-section&quot;&gt;
  &lt;summary id=&quot;31-you-can-run-your-first-quail-query-in-a-few-lines-of-code&quot;&gt;3.1 You can run your first Quail query in a few lines of code!&lt;span class=&quot;collapsible-note&quot;&gt;This part is collapsible, for space reasons.&lt;/span&gt;&lt;/summary&gt;

  &lt;p&gt;We can use Quail to run two AI filters over all 100,000 movie reviews in
the &lt;a href=&quot;https://huggingface.co/datasets/stanfordnlp/imdb&quot;&gt;Stanford IMDB
dataset&lt;/a&gt;. We first
download the reviews from Hugging Face, and load them into an Arrow
dataset.&lt;/p&gt;

  &lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;pyarrow&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pa&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;pyarrow.dataset&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ds&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;datasets&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;concatenate_datasets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;load_dataset&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;quail&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;imdb&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;load_dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;&quot;stanfordnlp/imdb&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;revision&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;e6281661ce1c48d982bc483cf8a173c1bbeb5d31&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;all_reviews&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;concatenate_datasets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;imdb&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;train&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;imdb&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;test&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;imdb&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;unsupervised&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;reviews&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ds&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;pa&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;({&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;&quot;review_id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pa&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;array&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sa&quot;&gt;f&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;review-&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;all_reviews&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))),&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;&quot;review&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;all_reviews&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;column&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;text&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;The query keeps reviews that discuss the movie’s ending, and recommend
watching the movie.&lt;/p&gt;

  &lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# This Python process has access to one H100.
&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;config&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;EngineConfig&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;gpus&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;qwen3-4b-fp8&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;backend&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;quail&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;h100-sxm&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;with&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;config&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;register&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;reviews&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;DocumentProvider&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;from_dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reviews&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;id_col&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;review_id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;n&quot;&gt;query&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sql&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&quot;&quot;
        SELECT r.review_id
        FROM reviews AS r
        WHERE AI.IF(
            PROMPT(
                &apos;Does this review discuss the ending of the movie?&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{0}&apos;,
                r.review
            ),
            -- Optional, but helps Quail reorder filters.
            {&apos;selectivity&apos;: 0.25}
        )
        AND AI.IF(
            PROMPT(
                &apos;Does the reviewer recommend watching the movie?&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{0}&apos;,
                r.review
            ),
            {&apos;selectivity&apos;: 0.5}
        )
    &quot;&quot;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dialect&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;bq&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;explain&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;run&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;table&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;collect&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;Before running the query, query.explain() prints the logical and
physical plans. The output below keeps only the parts that describe the
two filters and their execution settings.&lt;/p&gt;

  &lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;logical:
Project: r.review_id
SemanticFilter
predicate 1: discusses the ending (selectivity=25%)
predicate 2: recommends the movie (selectivity=50%)
Scan reviews as r [review, review_id]

physical: backend=quail, model=qwen3-4b-fp8, workers=1
KV=bf16
chunk budget=110,376 tokens
admission budget=362,250 tokens
Project: r.review_id (est. rows=12,500)
AiFilter: r (est. rows=12,500; est. time=124 s)
KV rewind=on
predicate 1 (input rows=100,000; est. pass=25%)
predicate 2 (input rows=25,000; est. pass=50%)
Scan reviews as r (rows=100,000)
tokens=29,926,924, mean_doc_tokens=299.3
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;The complete example in
&lt;a href=&quot;https://github.com/fsdatalab/quail/blob/d8d31f14f9d5c40d6cf683a74ba860788ce0d500/demos/imdb_ending_filter.py&quot;&gt;demos/imdb_ending_filter.py&lt;/a&gt;
prints the following results at the end of the run:&lt;/p&gt;

  &lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;matching reviews: 16057 of 100000
stage evaluated 100000 reviews, 0.283 passed
stage evaluated 28296 reviews, 0.568 passed
boot_s: 57.65 (cold)
token_wait_s: 0.0
wall_s: 277.41
total_s: 335.06 (boot + query)
fresh_tokens: 32499738
documents/second: 360.5
GPU price: $3.9492/GPU-hour (Modal)
GPU cost, including startup: $0.3675
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;The full run costs $0.3675 at &lt;a href=&quot;https://modal.com/pricing&quot;&gt;Modal’s H100
price&lt;/a&gt;, including model startup.&lt;sup&gt;&lt;a href=&quot;#note-5&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;
At current GPT-5 nano prices (including cached token prices), the same
two-filter workload would cost about $1.75, or 4.8 times the measured
Quail cost!&lt;sup&gt;&lt;a href=&quot;#note-6&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

  &lt;p&gt;&lt;strong&gt;Running on Modal&lt;/strong&gt;. If you don’t have a dedicated GPU, you can put the
whole query inside a Modal GPU function. The function creates a normal
Quail session and runs it:&lt;/p&gt;

  &lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;modal&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;app&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;modal&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;App&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;quail-engine&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;image&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;modal&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Image&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;from_registry&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;nvidia/cuda:13.0.1-devel-ubuntu24.04&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;add_python&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;3.12&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;entrypoint&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([])&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;uv_pip_install&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;quail-engine==0.1.0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;o&quot;&gt;@&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;app&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;function&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;image&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;image&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;gpu&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;H100!&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;memory&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32768&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;timeout&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1200&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;run_query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sql&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;documents&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;quail&lt;/span&gt;

    &lt;span class=&quot;n&quot;&gt;config&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;EngineConfig&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;gpus&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;qwen3-4b-fp8&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;backend&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;quail&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;h100-sxm&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;with&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;config&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;register&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
            &lt;span class=&quot;s&quot;&gt;&quot;docs&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;DocumentProvider&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;from_table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;documents&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;id_col&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;session&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sql&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sql&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;run&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;collect&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;report&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;Modal allocates the H100, and Quail plans and runs the query inside the
function. A complete example is in
&lt;a href=&quot;https://github.com/fsdatalab/quail/blob/d8d31f14f9d5c40d6cf683a74ba860788ce0d500/demos/quickstart_modal.py&quot;&gt;demos/quickstart_modal.py&lt;/a&gt;.&lt;/p&gt;

  &lt;p&gt;Check out the &lt;a href=&quot;https://fsdatalab.github.io/quail&quot;&gt;Quail documentation&lt;/a&gt;
to learn more.&lt;/p&gt;

&lt;/details&gt;

&lt;h3 id=&quot;32-quail-jointly-plans-queries-and-inference&quot;&gt;3.2 Quail jointly plans queries and inference.&lt;/h3&gt;

&lt;p&gt;This section describes Quail’s main design ideas at a high level. We are
still actively building Quail, and we will provide the full technical
details in a future report.&lt;/p&gt;

&lt;p&gt;We have three performance goals for Quail:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Minimize KV regret.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Keep the GPU busy by reducing CPU scheduling overhead.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Reach high model FLOP/s utilization (MFU) while the GPU is active.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Our current evaluation focuses on the first two goals. We defer a full
MFU study to future work.&lt;/p&gt;

&lt;p&gt;As shown in &lt;a href=&quot;#figure-3&quot;&gt;Figure 3&lt;/a&gt;, Quail consists of a query frontend, a query
planner, and an execution engine. Through the frontend, the user
provides Arrow tables or datasets, an AI-SQL or Python query, and the
model and GPU or GPUs to use. The frontend creates a logical plan from
the query. The query planner orders the filters and joins, chooses the
anchor for each join, and determines how many tokens each model forward
pass should process. The planner then lowers the logical plan into a
physical operator plan, which the execution engine runs.&lt;/p&gt;

&lt;figure class=&quot;figure-architecture&quot; id=&quot;figure-3&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/quail-architecture.svg&quot; alt=&quot;The Quail architecture.&quot; /&gt;
  &lt;figcaption&gt;Figure 3. Quail takes Arrow data and an AI-SQL query as input. The frontend creates a logical plan. The planner applies SQL rewrites, lowers the AI operations into physical operators, and plans their execution and KV reuse. The execution engine runs the physical plan and executes its AI operations on the GPU.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Quail is extensible, and its design is inspired by &lt;a href=&quot;https://datafusion.apache.org/&quot;&gt;Apache
DataFusion&lt;/a&gt;, an open source, extensible
analytical query engine. One can add new query
operators, planning rules, execution backends, models, or support for
other hardware.&lt;/p&gt;

&lt;h4 id=&quot;321-quail-turns-ai-sql-into-a-logical-query-plan&quot;&gt;3.2.1 Quail turns AI-SQL into a logical query plan.&lt;/h4&gt;

&lt;p&gt;Users register data as an in-memory Arrow table or an Arrow dataset.
Users can write queries in AI-SQL (we support both Snowflake’s and
BigQuery’s spellings, AI_FILTER and AI.IF), or use a Python query
builder similar to pandas. The current release of Quail supports AI
filters and joins, along with relational projections and LIMIT.&lt;/p&gt;

&lt;p&gt;Users define each &lt;a href=&quot;https://fsdatalab.github.io/quail/docs/user-guide/sql#ai-operators&quot;&gt;AI
operator&lt;/a&gt;
with a prompt and can provide optional planning information. The
optional selectivity gives the expected fraction of documents or
document pairs that will pass; without it, Quail keeps predicates in
their written order. For a join, the optional anchor chooses which input
comes first in the prompt for KV reuse; without it, the planner chooses
the anchor.&lt;/p&gt;

&lt;p&gt;Users can specify the model and GPU count. Quail currently supports
three models: Qwen3 4B FP8, Qwen3 32B FP8, and DiffusionGemma on H100
GPUs. We plan to add support for more models and hardware through the
extension interface.&lt;/p&gt;

&lt;p&gt;In Quail, each AI-SQL query is parsed with SQLGlot into a logical plan,
which is then passed to the query planner.&lt;/p&gt;

&lt;h4 id=&quot;322-quail-plans-operator-order-and-kv-reuse&quot;&gt;3.2.2 Quail plans operator order and KV reuse.&lt;/h4&gt;

&lt;p&gt;Database query optimizers already use many rules, e.g., pushing down
filters, reordering predicates, and choosing join order. AI-SQL adds several new
decisions, e.g., which document should be the join anchor, which KV will
be needed by a later operator, and how much model work should enter each
forward pass. Quail plans both kinds of decisions together.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Overview.&lt;/strong&gt; Given the logical query plan, we do the following:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;em&gt;Compute dataset statistics.&lt;/em&gt; We estimate the document lengths and
basic statistics for each input dataset.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;em&gt;Compute forward pass and KV limits.&lt;/em&gt; From the selected model and
GPU, we choose how many tokens to process in each model forward pass
and calculate the fixed KV capacity.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;em&gt;Perform SQL query rewrites&lt;/em&gt;. We push down projections and filters,
order the filters, and choose the join order and anchor for each
join.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;em&gt;Perform Inference-specific query rewrites.&lt;/em&gt; We lower AI operations
into physical operators and plan their execution order and KV use.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We describe these steps at a high level, in turn.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dataset statistics.&lt;/strong&gt; We estimate the row count, average document
length, and maximum document length for each input dataset.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Forward pass and KV limits.&lt;/strong&gt; Given the user’s selected model size and
GPU memory size, we calculate the maximum number of tokens to compute
for each model forward pass. We reserve HBM for the model weights and
two forward passes, and leave the rest of the HBM for the KV cache.
(This is more conservative than vLLM, which profiles one forward pass to
determine how much activation memory to reserve, so we can probably
improve on this).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SQL query rewrites&lt;/strong&gt;. We push projections and filters down to the
source datasets. We order filters using their estimated cost and
selectivity, following extremely well-known prior work (&lt;a href=&quot;https://dsf.berkeley.edu/jmh/miscpapers/sigmod93.pdf&quot;&gt;Hellerstein and
Stonebraker&lt;/a&gt; et
al.). For joins, we use a
&lt;a href=&quot;https://doi.org/10.1145/582095.582099&quot;&gt;Selinger-style&lt;/a&gt; search (i.e.,
System R) to choose the join order and anchor for each join. The cost
model uses the speed-of-light estimate from Section 2. We will explain
the calculation in a future post. For now, you can check out the &lt;a href=&quot;https://github.com/fsdatalab/quail/tree/main/quail/cost&quot;&gt;cost
model code&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Inference-specific query rewrites&lt;/strong&gt;. After the SQL rewrites, we
translate the logical plan into a DAG of physical operators. You can
find the physical operators that Quail currently supports in our
&lt;a href=&quot;https://fsdatalab.github.io/quail/docs/architecture/physical-plans&quot;&gt;physical plan
documentation&lt;/a&gt;.
For example, the AiFilter physical operator evaluates AI predicates over
documents, while AiJoin evaluates AI predicates over document pairs that
share an anchor. Each AI physical operator also specifies its prompt,
model, forward pass token budget, and KV settings (e.g., whether to
write KV to HBM because there will be a subsequent operator in the
query).&lt;/p&gt;

&lt;p&gt;Drawing inspiration from vectorized query execution, Quail streams
intermediate results directly between operators rather than
materializing complete datasets on disk or in main memory. For example,
in BIO-4, as soon as a batch of reports passes the initial filter
operator, Quail immediately pipelines it to the first join operator. It
maintains the report KV cache in HBM throughout both join operations,
allowing direct comparisons against the filtered reaction terms without
redundant KV recomputations.&lt;/p&gt;

&lt;h4 id=&quot;323-quail-runs-the-physical-query-plan&quot;&gt;3.2.3 Quail runs the physical query plan.&lt;/h4&gt;

&lt;p&gt;Overview. The execution engine has three main components:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;Physical plan executor.&lt;/strong&gt; On the CPU, Quail pulls document batches
through the physical operator DAG and prepares work for the GPU.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;KV manager.&lt;/strong&gt; Quail allocates, pins, “rewinds” (i.e., only
persists KV for the prefix we know will appear in a future operator,
not the entire LLM prompt which includes the document(s) and some
natural language instruction), and releases KV pages according to
the physical plan.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;Inference program.&lt;/strong&gt; On the GPU, Quail runs a model forward pass
for each input batch.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href=&quot;#figure-4&quot;&gt;Figure 4&lt;/a&gt; shows how these components work together.&lt;/p&gt;

&lt;figure class=&quot;figure-execution&quot; id=&quot;figure-4&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/quail-physical-plan.svg&quot; alt=&quot;The physical plan and execution path for BIO-4.&quot; /&gt;
  &lt;figcaption&gt;Figure 4. Quail lowers BIO-4 to the physical plan on the left. On the right, one CPU worker runs its physical operators and manages KV. Each AI physical operator invokes Quail’s inference program on the GPU.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;We’ll discuss the first two components; then we’ll describe the
inference program in &lt;a href=&quot;#324-quail-uses-specialized-inference-programs-for-ai-sql&quot;&gt;Section 3.2.4&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Physical plan executor.&lt;/strong&gt; Quail uses a pull-based executor, as in
&lt;a href=&quot;https://doi.org/10.1109/69.273032&quot;&gt;Volcano&lt;/a&gt;, but processes a batch at a
time, as in &lt;a href=&quot;https://www.cidrdb.org/cidr2005/papers/P19.pdf&quot;&gt;MonetDB&lt;/a&gt;.
Before execution, Quail tokenizes every document column referenced by an
AI filter or join with
&lt;a href=&quot;https://github.com/marcelroed/gigatoken&quot;&gt;Gigatoken&lt;/a&gt;&lt;sup&gt;&lt;a href=&quot;#note-7&quot;&gt;7&lt;/a&gt;&lt;/sup&gt;, then
loads one model copy per GPU. During execution, the CPU prepares one
input batch while the GPU processes another.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;KV manager.&lt;/strong&gt; Each GPU has a fixed pool of KV pages in HBM. After each
model evaluation, Quail retains only the KV that a later evaluation can
reuse. For a filter, Quail places the document before the
predicate-specific question. After the predicate returns TRUE or FALSE,
Quail discards the predicate KV and “rewinds” to the end of the document
KV. Then, if the predicate returns TRUE and another AI operator uses the
document, Quail will retain the “rewinded” KV in HBM; otherwise, Quail
will release it. Quail similarly retains “rewinded” KV for joins. If
eviction is necessary, Quail evicts the shortest documents, since longer
documents take disproportionately longer to recompute, thanks to
attention being a quadratic operation.&lt;/p&gt;

&lt;p&gt;Note that a general-purpose inference is different in that: (1) it
retains &lt;em&gt;all&lt;/em&gt; the KV associated with a request (no “rewinding”), even
though the suffix KV will never be used again in the query, (2) &lt;em&gt;all&lt;/em&gt;
requests’ KV are wastefully saved in HBM, even if documents are filtered
out in the query and never needed again, and (3) documents are evicted
with LRU.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Using multiple GPUs.&lt;/strong&gt; Our current multi-GPU support is quite basic.
We place one complete model copy and one KV pool on each GPU. We
partition filter documents and join anchors randomly and uniformly
across the GPUs, run them independently, and combine the results on the
CPU.&lt;/p&gt;

&lt;h4 id=&quot;324-quail-uses-specialized-inference-programs-for-ai-sql&quot;&gt;3.2.4 Quail uses specialized inference programs for AI-SQL.&lt;/h4&gt;

&lt;p&gt;During planning, Quail chooses which documents or document pairs require
model evaluation. During execution, each evaluation follows an
&lt;em&gt;inference program&lt;/em&gt;: e.g., embedding lookup, transformer layers,
attention, matrix multiplication, etc.&lt;/p&gt;

&lt;p&gt;Here, we first describe how physical operators are expressed as
inference programs, then how vLLM represents an inference program (which
we adopt), and finally, the changes we make to Quail’s inference
program.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Physical operator interface.&lt;/strong&gt; Each AiFilter or AiJoin physical
operator is expressed as an inference program. The program takes token
IDs and positions, plus the locations of any reusable KV pages. It
returns TRUE and FALSE scores for each row or pair of rows. Quail runs
the program across all rows or pairs evaluated by the operator.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;vLLM’s inference programs.&lt;/strong&gt; vLLM is a general-purpose engine designed
to support any inference pattern, across various model architectures and
hardware backends. How does vLLM &lt;em&gt;do it all&lt;/em&gt;? As shown in &lt;a href=&quot;#figure-5&quot;&gt;Figure 5&lt;/a&gt;,
given the model choice and GPU, there are two primary paths through
which vLLM creates an inference program (i.e., of GPU kernels): (1)
PyTorch operations JIT-compiled with
&lt;a href=&quot;https://docs.vllm.ai/en/stable/design/torch_compile/&quot;&gt;torch.compile&lt;/a&gt;
and TorchInductor into generated Triton GPU kernels, and (2) custom
operations (such as attention) expressed through highly specialized GPU
kernels like, FlashAttention.&lt;/p&gt;

&lt;figure class=&quot;figure-medium&quot; id=&quot;figure-5&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/vllm-kernel-selection.svg&quot; alt=&quot;How vLLM builds a model forward pass.&quot; /&gt;
  &lt;figcaption&gt;Figure 5. vLLM builds the model’s forward pass for the selected model and GPU through two paths: compiling PyTorch operations into GPU kernels and selecting prewritten kernels for specialized operations.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Quail’s inference program.&lt;/strong&gt; We did not reimplement every model and
GPU operation from scratch. That would be silly. Instead, Quail uses
vLLM’s model implementations to obtain the operations required for a
forward pass, then runs them with its own scheduler and KV manager.
However, Quail makes three small changes to the forward pass:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First, fuse small operations.&lt;/strong&gt; We write
&lt;a href=&quot;https://triton-lang.org/&quot;&gt;Triton&lt;/a&gt; kernels that fuse normalization with
FP8 quantization, Q/K normalization with RoPE, and activation with FP8
quantization. This is extremely easy to do now with AI agents; it
requires no novel kernel design ideas. By fusing these operations, Quail
reduces kernel launches and intermediate HBM traffic.&lt;sup&gt;&lt;a href=&quot;#note-8&quot;&gt;8&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second, specialize attention for joins.&lt;/strong&gt; An AI join compares one
anchor with many partners. Standard vLLM treats each anchor and partner
as a separate sequence. Attention therefore reads the same anchor KV
again for every partner, as the left side of &lt;a href=&quot;#figure-6&quot;&gt;Figure 6&lt;/a&gt; shows.&lt;/p&gt;

&lt;figure class=&quot;figure-medium&quot; id=&quot;figure-6&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/join-anchor-kv-reads.svg&quot; alt=&quot;Anchor KV reads in vLLM and Quail for an AI join.&quot; /&gt;
  &lt;figcaption&gt;Figure 6. In vLLM, each anchor and partner pair is a separate sequence, so attention reads the same anchor KV once per partner. Quail groups the partners that share an anchor, so attention reads the anchor KV once.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Quail groups all partners that share an anchor and computes the anchor
KV once (right side of &lt;a href=&quot;#figure-6&quot;&gt;Figure 6&lt;/a&gt;). &lt;a href=&quot;#figure-7&quot;&gt;Figure 7&lt;/a&gt; shows how Quail evaluates attention in two parts. One
&lt;a href=&quot;https://arxiv.org/abs/2407.08608&quot;&gt;FlashAttention 3&lt;/a&gt; call
computes causal attention within each partner. A second call applies all
partner queries to the shared anchor KV, reducing repeated reads. Quail
combines the two results using their log-sum-exp values and the
&lt;a href=&quot;https://arxiv.org/abs/1805.02867&quot;&gt;online softmax formula&lt;/a&gt;,
producing the same output as attention over each full anchor and partner
sequence. This is one level of “tree”-based attention.&lt;sup&gt;&lt;a href=&quot;#note-9&quot;&gt;9&lt;/a&gt;&lt;/sup&gt; One
Triton kernel combines the BF16 outputs and converts them to the FP8
format expected by the output projection.&lt;/p&gt;

&lt;figure class=&quot;figure-join-attention&quot; id=&quot;figure-7&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/quail-join-attention.svg&quot; alt=&quot;How Quail evaluates attention for an AI join.&quot; /&gt;
  &lt;figcaption&gt;Figure 7. Quail evaluates join attention with two FlashAttention 3 calls. One computes attention within each partner suffix. The other applies the suffix queries to the shared anchor KV. A Triton kernel combines both results and converts the output to FP8 before the output projection.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;strong&gt;Third, restrict the output head to&lt;/strong&gt; TRUE &lt;strong&gt;and&lt;/strong&gt; FALSE. Normally, a
model would use its final output head (“language modeling” head,
lm_head) to compute a score for every token in its vocabulary. For AI
filters and joins, Quail needs only the scores for token IDs that
represent TRUE or FALSE.&lt;sup&gt;&lt;a href=&quot;#note-10&quot;&gt;10&lt;/a&gt;&lt;/sup&gt; Quail therefore multiplies the
final hidden state by only the corresponding rows of the output/language
modeling head matrix. By using the smaller matrix, Quail reduces
computation and GPU memory use by the output head.&lt;/p&gt;

&lt;h2 id=&quot;4-we-evaluate-quail-against-vllm&quot;&gt;4. We evaluate Quail against vLLM.&lt;/h2&gt;

&lt;p&gt;At scale factor 0.1, Quail is faster than a “stock” vLLM baseline on 27
of the 29 QUAIL-B queries. The &lt;strong&gt;(geometric) mean speedup is 1.84x&lt;/strong&gt;, and
the &lt;strong&gt;maximum speedup is 11.22x&lt;/strong&gt; on BIO-2. At scale factor 1.0, we find
a query for which Quail is 14.04x faster! The two queries where stock
vLLM wins expose one missing feature clearly: Quail does not yet reuse
matching prefixes across different rows.&lt;/p&gt;

&lt;p&gt;In this section, we first describe our &lt;a href=&quot;#41-metrics-and-baselines-for-ai-sql-performance&quot;&gt;metrics and
baselines&lt;/a&gt;, then
present the &lt;a href=&quot;#42-overall-quail-is-184x-faster-across-quail-b&quot;&gt;full QUAIL-B results&lt;/a&gt;,
and finally examine &lt;a href=&quot;#43-quail-dominates-vllm-on-bio-4-1404x-faster&quot;&gt;BIO-4&lt;/a&gt;
and &lt;a href=&quot;#44-but-vllm-dominates-quail-on-agent-1-quail-takes-232x-as-long&quot;&gt;AGENT-1&lt;/a&gt;
in detail.&lt;/p&gt;

&lt;h3 id=&quot;41-metrics-and-baselines-for-ai-sql-performance&quot;&gt;4.1 Metrics and baselines for AI-SQL performance.&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Metrics.&lt;/strong&gt; We report three metrics for each query: KV regret,
$/query, and input tokens/second. KV regret is repeated model work:
fresh input tokens beyond the minimum needed to compute each reusable
prefix once. $/query is query runtime in hours multiplied by
&lt;a href=&quot;https://modal.com/pricing&quot;&gt;$3.9492 per H100-hour&lt;/a&gt;. Input
tokens/second is the total requested input tokens divided by query
runtime. Each evaluated prompt contributes its full input length,
including tokens served from KV. Lower KV regret and cost are better;
higher throughput is better.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;QUAIL-B.&lt;/strong&gt; We created
&lt;a href=&quot;https://github.com/fsdatalab/quail-bench&quot;&gt;QUAIL-B&lt;/a&gt;, a benchmark
with 29 AI-SQL queries. It covers IMDB reviews, medical reports,
fact-checking claims, legal documents, and software-agent traces.
Queries include filters, filter sequences, and one or more joins. Each
dataset has scale factors 0.1, 0.5, and 1.0. We compare all 29 default
queries at scale factor 0.1. We examine BIO-4 at scale factor 1.0 and
AGENT-1 in more detail.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Setup.&lt;/strong&gt; Every query uses Qwen3 4B FP8 with BF16 KV on one H100. Quail
and each vLLM baseline run one after the other on the same physical GPU.
They use the same model, prompts, and logical query plan.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;vLLM baselines.&lt;/strong&gt; Both baselines use the optimal operator ordering
chosen by our query planner. For each operator, we prepare its requests
and order them to improve KV reuse. We call the operator-at-a-time
baseline “stock vLLM.” For QUAIL-B queries with multiple filters or
joins, we also report a “pipelined vLLM” baseline. It pipelines requests
between consecutive filters and between consecutive joins. For fairness,
both baselines use Gigatoken for tokenization, as Quail does, instead of
vLLM’s Hugging Face tokenizer.&lt;sup&gt;&lt;a href=&quot;#note-11&quot;&gt;11&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h3 id=&quot;42-overall-quail-is-184x-faster-across-quail-b&quot;&gt;4.2 Overall, Quail is 1.84x faster across QUAIL-B.&lt;/h3&gt;

&lt;p&gt;&lt;a href=&quot;#table-1&quot;&gt;Table 1&lt;/a&gt; averages tokens/second, KV regret, and cost per query within
each dataset at scale factor 0.1. Cost multipliers are relative to Quail.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-1&quot;&gt;
&lt;table&gt;
&lt;colgroup&gt;
&lt;col style=&quot;width: 33%&quot; /&gt;
&lt;col style=&quot;width: 33%&quot; /&gt;
&lt;col style=&quot;width: 33%&quot; /&gt;
&lt;/colgroup&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;th&gt;Dataset&lt;/th&gt;
&lt;th&gt;Quail&lt;/th&gt;
&lt;th&gt;Stock vLLM&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;BIO (4 queries)&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;12,296,410 tokens/s&lt;/p&gt;
&lt;p&gt;157,995 KV regret&lt;/p&gt;
&lt;p&gt;$0.0892/query (1.00x)&lt;/p&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;1,420,421 tokens/s&lt;/p&gt;
&lt;p&gt;1,441,816 KV regret&lt;/p&gt;
&lt;p&gt;$0.7771/query (8.72x)&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;IMDB (10 queries)&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;649,864 tokens/s&lt;/p&gt;
&lt;p&gt;451,917 KV regret&lt;/p&gt;
&lt;p&gt;$0.0282/query (1.00x)&lt;/p&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;382,818 tokens/s&lt;/p&gt;
&lt;p&gt;1,129,174 KV regret&lt;/p&gt;
&lt;p&gt;$0.0467/query (1.66x)&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;FEV (8 queries)&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;1,692,276 tokens/s&lt;/p&gt;
&lt;p&gt;408,592 KV regret&lt;/p&gt;
&lt;p&gt;$0.0383/query (1.00x)&lt;/p&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;724,574 tokens/s&lt;/p&gt;
&lt;p&gt;519,588 KV regret&lt;/p&gt;
&lt;p&gt;$0.0820/query (2.14x)&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;LEP (5 queries)&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;388,903 tokens/s&lt;/p&gt;
&lt;p&gt;2,147 KV regret&lt;/p&gt;
&lt;p&gt;$0.0670/query (1.00x)&lt;/p&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;316,617 tokens/s&lt;/p&gt;
&lt;p&gt;70,982 KV regret&lt;/p&gt;
&lt;p&gt;$0.0853/query (1.27x)&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;AGENT (2 queries)&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;73,006 tokens/s&lt;/p&gt;
&lt;p&gt;11,886,152 KV regret&lt;/p&gt;
&lt;p&gt;$0.2616/query (1.00x)&lt;/p&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: left;&quot;&gt;&lt;p&gt;169,201 tokens/s&lt;/p&gt;
&lt;p&gt;23,928 KV regret&lt;/p&gt;
&lt;p&gt;$0.1129/query (0.43x)&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p class=&quot;table-caption&quot;&gt;Table 1. Average QUAIL-B results by dataset at scale factor 0.1. Cost multipliers are relative to Quail.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;&lt;a href=&quot;#figure-8&quot;&gt;Figure 8&lt;/a&gt; summarizes throughput by dataset, and &lt;a href=&quot;#figure-9&quot;&gt;Figure 9&lt;/a&gt; reports latency
for all 29 queries. The geometric mean of Quail’s per-query speedups
over stock vLLM is 1.84x. In total, Quail completes the benchmark in
1,643.74 seconds, compared with 4,451.95 seconds for stock vLLM. Quail
takes 3.35x longer than the combined SoL estimate of 491.17 seconds, so
there is substantial room to improve.&lt;/p&gt;

&lt;figure class=&quot;figure-plot-compact&quot; id=&quot;figure-8&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/quail-throughput-by-dataset.png&quot; alt=&quot;Quail throughput by QUAIL-B dataset.&quot; /&gt;
  &lt;figcaption&gt;Figure 8. Average requested input tokens per second on QUAIL-B, shown as a percentage of the Speed-of-Light estimate (i.e., theoretical hardware limits) for each dataset. We use Qwen3 4B FP8 and one H100.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;figure id=&quot;figure-9&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/quail-query-latency.png&quot; alt=&quot;Latency for all 29 QUAIL-B queries.&quot; /&gt;
  &lt;figcaption&gt;Figure 9. Query latency at scale factor 0.1. Bars show Quail and stock vLLM; horizontal lines show SoL estimates. The vertical axis uses a log scale because the query times span more than three orders of magnitude.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;&lt;a href=&quot;#figure-10&quot;&gt;Figure 10&lt;/a&gt; focuses on the eight queries where pipelining changes how vLLM
submits requests. Pipelined vLLM is faster than stock vLLM on seven of
them, by 1.12x on average and up to 1.27x on IMDB-6.&lt;/p&gt;

&lt;figure class=&quot;figure-plot-compact&quot; id=&quot;figure-10&quot;&gt;
  &lt;img src=&quot;/assets/blog/introducing-quail/vllm-pipelining-throughput.png&quot; alt=&quot;Stock and pipelined vLLM throughput on eight QUAIL-B queries.&quot; /&gt;
  &lt;figcaption&gt;Figure 10. Requested input tokens per second as a percentage of each query’s SoL estimate. Stock vLLM finishes one filter stage before submitting the next. Pipelined vLLM submits the next filter for each document as soon as the previous filter returns TRUE.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;Stock vLLM is faster than Quail only on AGENT-1 and AGENT-2. Quail does
not yet reuse matching prefixes across rows, so it recomputes far more
KV tokens on each query. Section 4.4 examines AGENT-1.&lt;/p&gt;

&lt;h3 id=&quot;43-quail-dominates-vllm-on-bio-4-1404x-faster&quot;&gt;4.3 Quail dominates vLLM on BIO-4: 14.04x faster!&lt;/h3&gt;

&lt;p&gt;BIO-4 contains the kind of reuse Quail currently handles well: long
shared documents, two joins, and millions of related model calls whose
order is known before execution. At scale factor 1.0, BIO-4 filters
5,000 medical reports and two uses of the same 4,144 reaction terms,
then runs two joins over the surviving inputs.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#table-2&quot;&gt;Table 2&lt;/a&gt; reports throughput, cost, and KV regret.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-2&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th style=&quot;text-align: left&quot;&gt;Metric&lt;/th&gt;
        &lt;th&gt;Quail&lt;/th&gt;
        &lt;th&gt;Stock vLLM&lt;/th&gt;
        &lt;th&gt;SoL estimate&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;Requested input tokens/s&lt;/td&gt;
        &lt;td&gt;19.03 million&lt;/td&gt;
        &lt;td&gt;1.36 million&lt;/td&gt;
        &lt;td&gt;37.37 million&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;GPU cost per query&lt;/td&gt;
        &lt;td&gt;$1.93&lt;/td&gt;
        &lt;td&gt;$27.03&lt;/td&gt;
        &lt;td&gt;$0.98&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;KV regret&lt;/td&gt;
        &lt;td&gt;18.0 million&lt;/td&gt;
        &lt;td&gt;50.3 million&lt;/td&gt;
        &lt;td&gt;0 (assumed)&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 2. BIO-4 results at scale factor 1.0. GPU cost excludes model startup. SoL values are estimates.&lt;/p&gt;

&lt;/div&gt;

&lt;p&gt;Quail takes 29.26 minutes, compared with 6.84 hours for stock vLLM.
Quail is 14.04x faster. It is 1.96x the SoL estimate, while stock vLLM
is 27.55x the estimate. Even with pipelining, vLLM still takes 4.00
hours.&lt;/p&gt;

&lt;p&gt;Quail costs $1.93 per query, compared with $27.03 for stock vLLM. The
SoL cost estimate is $0.98 per query. Quail processes 19.03 million
requested input tokens per second, compared with 1.36 million for stock
vLLM.&lt;/p&gt;

&lt;p&gt;Quail also recomputes less KV. It recomputes 18.0 million tokens,
compared with 50.3 million for stock vLLM.&lt;/p&gt;

&lt;h3 id=&quot;44-but-vllm-dominates-quail-on-agent-1-quail-takes-232x-as-long&quot;&gt;4.4 But, vLLM dominates Quail on AGENT-1: Quail takes 2.32x as long.&lt;/h3&gt;

&lt;p&gt;AGENT-1 contains a different kind of reuse. It filters 1,772 cumulative
snapshots from software agent runs. Separate rows contain overlapping
prefixes from the same agent trace, and stock vLLM’s automatic prefix
caching recognizes them. Quail does not yet recognize that relationship,
so stock vLLM wins. &lt;a href=&quot;#table-3&quot;&gt;Table 3&lt;/a&gt; shows two example rows.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-3&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th style=&quot;text-align: left&quot;&gt;id&lt;/th&gt;
        &lt;th&gt;trajectory_id&lt;/th&gt;
        &lt;th&gt;turn_index&lt;/th&gt;
        &lt;th&gt;trace&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;trace_42_turn_5&lt;/td&gt;
        &lt;td&gt;trace_42&lt;/td&gt;
        &lt;td&gt;5&lt;/td&gt;
        &lt;td&gt;[USER] Fix the failing parser. [ASSISTANT] Tries approach A. [TOOL] The test fails.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;trace_42_turn_10&lt;/td&gt;
        &lt;td&gt;trace_42&lt;/td&gt;
        &lt;td&gt;10&lt;/td&gt;
        &lt;td&gt;&amp;lt;complete trace from turn 5&amp;gt; [ASSISTANT] Finds the mistake, and tries approach B. [TOOL] The tests pass.&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 3. Two cumulative snapshots from the same software agent trace.&lt;/p&gt;

&lt;/div&gt;

&lt;p&gt;Here is the AGENT-1 query, simplified for this post:&lt;/p&gt;

&lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;agent_traces&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;WHERE&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;Did the agent recover after trying an approach that did not work?&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n\n&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;{0}&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;trace&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href=&quot;#table-4&quot;&gt;Table 4&lt;/a&gt; reports throughput, cost, and KV regret.&lt;/p&gt;

&lt;div class=&quot;table-wrap&quot; id=&quot;table-4&quot;&gt;

  &lt;table&gt;
    &lt;thead&gt;
      &lt;tr&gt;
        &lt;th style=&quot;text-align: left&quot;&gt;Metric&lt;/th&gt;
        &lt;th&gt;Quail&lt;/th&gt;
        &lt;th&gt;Stock vLLM&lt;/th&gt;
        &lt;th&gt;SoL estimate&lt;/th&gt;
      &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;Requested input tokens/s&lt;/td&gt;
        &lt;td&gt;73,006&lt;/td&gt;
        &lt;td&gt;169,201&lt;/td&gt;
        &lt;td&gt;367,400&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;GPU cost per query&lt;/td&gt;
        &lt;td&gt;$0.2623&lt;/td&gt;
        &lt;td&gt;$0.1131&lt;/td&gt;
        &lt;td&gt;$0.0521&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td style=&quot;text-align: left&quot;&gt;KV regret&lt;/td&gt;
        &lt;td&gt;11,886,152&lt;/td&gt;
        &lt;td&gt;23,928&lt;/td&gt;
        &lt;td&gt;0 (assumed)&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/tbody&gt;
  &lt;/table&gt;

  &lt;p class=&quot;table-caption&quot;&gt;Table 4. AGENT-1 results. GPU cost excludes model startup. SoL values are estimates.&lt;/p&gt;

&lt;/div&gt;

&lt;p&gt;Stock vLLM finishes AGENT-1 in 103.07 seconds, compared with Quail’s
239.12 seconds. Quail takes 2.32x as long. The SoL estimate is 47.47 seconds, so stock vLLM still
takes 2.17x longer than the estimate.&lt;/p&gt;

&lt;p&gt;Stock vLLM wins because its automatic prefix caching can reuse KV across
rows with matching token prefixes. Quail currently reuses KV only when
the same document appears again in the query, not across different
documents. As a result, Quail incurs 11.89 million KV regret tokens,
while stock vLLM incurs only 23,928.&lt;/p&gt;

&lt;p&gt;We plan to add automatic prefix caching to Quail, but the lookup must
remain cheap at the request volumes that AI-SQL queries can produce.&lt;/p&gt;

&lt;h2 id=&quot;5-put-another-way-quail-brings-jev-like-speeds-and-intelligence-to-database-scale-workloads&quot;&gt;5. Put another way: Quail brings Jev-like speeds and intelligence to database-scale workloads.&lt;/h2&gt;

&lt;p&gt;Quail also supports &lt;a href=&quot;https://huggingface.co/RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic&quot;&gt;DiffusionGemma 26B-A4B
FP8&lt;/a&gt;,
a larger mixture-of-experts model with 4B active parameters per token.
This gives Quail a higher-intelligence option that is still extremely
fast. On IMDB-2 at scale factor 0.1, DiffusionGemma matched 88.89% of
Qwen3 32B’s answers, compared with 76.41% for Qwen3 4B. It ran the query
in 32.41 seconds, or 1.53x as long as Qwen3 4B’s 21.20 seconds, on one H100.&lt;/p&gt;

&lt;p&gt;This fits a broader class of workloads that need fast, bounded model
decisions instead of long generated responses.
&lt;a href=&quot;https://typesafe.ai/blog/introducing-system-one-models-and-jev&quot;&gt;Jev&lt;/a&gt;
has highlighted the demand for this pattern in application backends.
Quail targets its batch, online analytical processing (OLAP) version:
one query creates thousands or millions of related decisions over a
dataset, and Quail plans and runs them together. This makes Quail a good
fit for LLM judge workflows, trace compaction, labeling, and other
large-scale data transformations.&lt;/p&gt;

&lt;h2 id=&quot;6-we-are-just-getting-started-with-quail&quot;&gt;6. We are just getting started with Quail!&lt;/h2&gt;

&lt;p&gt;Some next steps are obvious. E.g., we want to support more AI-SQL
operators, more models, and more hardware. We especially want to support
tiny hybrid models, so you can run Quail on a MacBook.&lt;/p&gt;

&lt;p&gt;We are also interested many research ideas fusing the database and
inference worlds; here are just a few:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use the full memory hierarchy for KV.&lt;/strong&gt; Quail currently keeps reusable
KV in GPU HBM or recomputes it. We want to move KV to host memory or
local SSD when it does not fit on the GPU, then bring it back before
reuse. We also want automatic prefix caching across rows. Perhaps we
will also want to do KV compression (we know it is good to make indexes
smaller).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Improve model FLOP/S utilization.&lt;/strong&gt; Quail currently relies on DeepGEMM
and FlashAttention for its main GPU kernels. We have not optimized the
kernels themselves, and we are stoked to be working with Modal and
Doubleword, inference experts, on kernel optimization.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Train models for execution and planning.&lt;/strong&gt; &lt;a href=&quot;https://arxiv.org/abs/2603.15970&quot;&gt;Google’s work on
lightweight proxy models for
AI-SQL&lt;/a&gt; suggests small models can
evaluate filters cheaply. The same models could predict selectivity and
likely survivors for the query planner, helping Quail choose operator
order and decide which KV to keep. The big systems question is how to
run and train many specialized models alongside a larger model on the
same GPU.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can an AI join work like a hash join?&lt;/strong&gt; Today, Quail reuses an anchor’s KV within
one join loop, but it recomputes every partner for each new anchor. The
same document is therefore encoded once per pair. Could we instead
encode every document once and use its KV as a position-independent
index entry? A document from the other relation could then search those
entries for matches, like probing a hash table, without recomputing the
indexed documents. This may require removing or separating the position
information that RoPE adds to KV.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;New methods for using AI agents to build systems.&lt;/strong&gt; We used AI coding
agents heavily to build the current version of Quail. We expect to keep using agents to build many of
the features above. How do we do this correctly? We want better ways to specify what the system must do, and to
check that every agent-written change keeps answers correct and runtime
close to SoL. It feels inevitable that agents will do the bulk of the
coding, and we are excited to build Quail in public and share the
meta-learnings from building it with agents.&lt;/p&gt;

&lt;p&gt;More blog posts, and eventually a technical report, are coming soon. For
now, please try &lt;a href=&quot;https://github.com/fsdatalab/quail&quot;&gt;Quail&lt;/a&gt;! If these ideas sound interesting, reach out to
get involved! And if you want to build an application on top of Quail,
such as an LLM judge workflow in AI-SQL, Quail has an MIT license. It is
now orders of magnitude cheaper to add intelligence to your data
processing workflows, and we would love to see what you build :-)&lt;/p&gt;

&lt;h2 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h2&gt;

&lt;p&gt;We thank &lt;a href=&quot;https://modal.com/&quot;&gt;Modal&lt;/a&gt; for sponsoring the compute used in
this research.&lt;/p&gt;

&lt;h2 id=&quot;notes&quot;&gt;Notes&lt;/h2&gt;

&lt;p&gt;&lt;span id=&quot;note-1&quot;&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/span&gt; The query is based on the &lt;a href=&quot;https://aclanthology.org/2023.findings-emnlp.896/&quot;&gt;BioDEX
dataset&lt;/a&gt;. The SQL form
of BIO-4 is shown below. In each prompt, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{0}&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{1}&lt;/code&gt; refer to the first
and second arguments.&lt;/p&gt;

&lt;div class=&quot;note-1-sql&quot;&gt;

  &lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;SELECT&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;neurological_reaction_id&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;id&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cardiovascular_reaction_id&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;FROM&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reports&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;JOIN&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reaction_terms&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;ON&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s1&quot;&gt;&apos;Does the medical report in {0} describe the reaction in {1} as &apos;&lt;/span&gt;
        &lt;span class=&quot;s1&quot;&gt;&apos;something the patient experienced?&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;report&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;term&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;JOIN&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reaction_terms&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;AS&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;c&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;ON&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
        &lt;span class=&quot;s1&quot;&gt;&apos;Does the medical report in {0} describe the reaction in {1} as &apos;&lt;/span&gt;
        &lt;span class=&quot;s1&quot;&gt;&apos;something the patient experienced?&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;report&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;term&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;WHERE&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;Does {0} describe a serious or life-threatening adverse event?&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;report&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;AND&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;Is this reaction neurological, affecting the nervous system? {0}&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;term&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;AND&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;Is this reaction cardiovascular, affecting the heart or blood vessels? {0}&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;term&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

&lt;/div&gt;

&lt;p&gt;&lt;span id=&quot;note-2&quot;&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/span&gt; Prompts use numbered placeholders, such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{0}&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{1}&lt;/code&gt;, to refer to
the arguments after the prompt string in the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PROMPT&lt;/code&gt; call. The SQL call
and the model input it produces are shown below.&lt;/p&gt;

&lt;div class=&quot;note-2-example&quot;&gt;

  &lt;div class=&quot;language-sql highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;AI&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;IF&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PROMPT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;Does {0} mention {1}?&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;report&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;term&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;The documents do not have to appear exactly where their placeholders
occur in the question. Quail can place the report first so its KV can be
reused when the same report is compared with another reaction term.&lt;/p&gt;

  &lt;p&gt;The model receives the following input:&lt;/p&gt;

  &lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;DOCUMENT:
[contents of r.report]

(The document above is DOCUMENT {0}.)

Evaluate TRUE or FALSE for the following question:
Does {0} mention {1}?

DOCUMENT {1}:
[contents of n.term]
ANSWER:
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

  &lt;p&gt;Of course, whether other prompt layouts affect accuracy remains an open
question, though we expect this to matter less as models improve.&lt;/p&gt;

&lt;/div&gt;

&lt;p&gt;&lt;span id=&quot;note-3&quot;&gt;&lt;strong&gt;3.&lt;/strong&gt;&lt;/span&gt; The speed of light estimate assumes 100 percent model FLOP/s utilization
(MFU), meaning every forward pass sustains the GPU’s peak arithmetic
throughput. Real systems cannot reach that rate, so the estimate is an
optimistic lower bound.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-4&quot;&gt;&lt;strong&gt;4.&lt;/strong&gt;&lt;/span&gt; Modal provides useful background on &lt;a href=&quot;https://modal.com/blog/gpu-utilization-guide&quot;&gt;GPU
utilization&lt;/a&gt; and
&lt;a href=&quot;https://modal.com/blog/host-overhead-inference-efficiency&quot;&gt;host
overhead&lt;/a&gt;
in inference engines.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-5&quot;&gt;&lt;strong&gt;5.&lt;/strong&gt;&lt;/span&gt; The IMDB dataset was already on disk, so the measurement excludes
the time and cost of downloading it.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-6&quot;&gt;&lt;strong&gt;6.&lt;/strong&gt;&lt;/span&gt; We use OpenAI’s &lt;a href=&quot;https://developers.openai.com/api/docs/models/gpt-5-nano&quot;&gt;cached-token
price&lt;/a&gt; in this
estimate and assume an “infinite” cache, so every reusable document token
receives that rate.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-7&quot;&gt;&lt;strong&gt;7.&lt;/strong&gt;&lt;/span&gt; Marcel Rød built the fast
&lt;a href=&quot;https://github.com/marcelroed/gigatoken&quot;&gt;Gigatoken&lt;/a&gt; tokenizer;
thank you!&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-8&quot;&gt;&lt;strong&gt;8.&lt;/strong&gt;&lt;/span&gt; Kernel fusion can substantially improve prefill MFU. In &lt;a href=&quot;https://www.sailresearch.com/blog/tpu-v6e-gemma&quot;&gt;“Chasing
Speed of Light on TPU
v6e,”&lt;/a&gt; Sail
Research reports increasing Gemma 4 31B prefill MFU from about 32
percent to 63 percent through several optimizations, including folding
activation, normalization, and RoPE work into surrounding kernels.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-9&quot;&gt;&lt;strong&gt;9.&lt;/strong&gt;&lt;/span&gt; We follow a long line of “Tree”-based attention approaches, which
evaluate several branches that share a prefix without allowing one
branch to attend to another. E.g.,
&lt;a href=&quot;https://arxiv.org/abs/2305.09781&quot;&gt;SpecInfer&lt;/a&gt; uses a tree mask
during speculative decoding to verify several possible continuations at
once. Also, &lt;a href=&quot;https://arxiv.org/abs/2402.05099&quot;&gt;Hydragen&lt;/a&gt; uses
shared-prefix attention during decoding to generate several outputs from
one input. Quail applies the same structure, but during prefill.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-10&quot;&gt;&lt;strong&gt;10.&lt;/strong&gt;&lt;/span&gt; One might expect two token IDs, one for each answer. For Qwen,
Quail scores four spellings of each answer. The TRUE tokens are “TRUE”
(20611), “&lt;span class=&quot;space-symbol&quot;&gt;␠&lt;/span&gt;TRUE” (8214), “True” (2514), and “&lt;span class=&quot;space-symbol&quot;&gt;␠&lt;/span&gt;True” (3007). The FALSE
tokens are “FALSE” (30351), “&lt;span class=&quot;space-symbol&quot;&gt;␠&lt;/span&gt;FALSE” (7833), “False” (4049), and
“&lt;span class=&quot;space-symbol&quot;&gt;␠&lt;/span&gt;False” (3557). Here, &lt;span class=&quot;space-symbol&quot;&gt;␠&lt;/span&gt; marks a leading space.&lt;/p&gt;

&lt;p&gt;&lt;span id=&quot;note-11&quot;&gt;&lt;strong&gt;11.&lt;/strong&gt;&lt;/span&gt; Both baselines use vLLM 0.26.0 with automatic prefix caching. We use
the largest stable settings: 25,305 maximum batched tokens, 4,096
sequences, GPU memory utilization of 0.91, and one CUDA graph for 8,192
tokens. Larger settings ran out of GPU memory.&lt;/p&gt;

&lt;h2 id=&quot;cite-this-post&quot;&gt;Cite this post&lt;/h2&gt;

&lt;div class=&quot;bibtex-block&quot;&gt;
  &lt;p&gt;&lt;button class=&quot;copy-bibtex&quot; type=&quot;button&quot;&gt;Copy BibTeX&lt;/button&gt;&lt;/p&gt;

  &lt;div class=&quot;language-bibtex highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nc&quot;&gt;@misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;shankar2026quail&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;title&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Building an Ultra-High Throughput AI-SQL Engine}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;author&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{Shankar, Shreya and Frye, Charles and Finn, Fergus and
            Dhariya, Arnav and Barrow, Joseph and Arik, Meryem}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;year&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{2026}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;month&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;sep&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;url&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;{https://fsdatalab.github.io/blog/introducing-quail/}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;

&lt;/div&gt;
</content>
  </entry>
</feed>
