Execution

Quail executes the operators selected by the physical plan. The CPU coordinates their inputs and outputs. The GPU performs model inference.

Quail can ask several questions about a document without processing the whole document again for each question. See Physical plans for an example plan.

Operator scheduling

The runner

The runner coordinates the operators in the plan. It runs an operator when the operator's required inputs are available.

• Model operators, such as classification, use the GPU.
• Operations that test computed labels or select result columns use Apache Arrow on the CPU.
• Compatible operators can pass documents to one another before they finish processing the whole input table.

Pipelines and batches

A pipeline is a group of operators that can pass documents to one another before they finish processing the whole input table. KV is the key and value vectors saved for tokens the model has already processed. Quail can keep a document's KV available as it passes through the pipeline.

For example, imagine a reviews table containing review text. A query first keeps reviews that discuss the acting, then classifies their sentiment, then keeps negative reviews. In a pipeline:

1. A review reaches classification as soon as it passes the first filter.
2. Classification reuses the review's KV and adds the sentiment question.
3. The CPU tests the returned label and keeps the review if it is negative.

A model request is the model computation needed for one document or pair at one stage of execution. A batch combines model requests that can be processed together. Requests from different pipeline stages can share a batch.

A forward pass processes a batch's new tokens through the model, using any saved KV. The CPU prepares the next batch while the GPU processes the current batch.

Quail limits each batch by token count and available KV memory:

• The token budget limits how many new token positions a forward pass can process. Shorter requests allow more documents to fit in a batch.
• The KV budget limits the GPU memory used for saved KV. Long documents can fill that memory before the token budget is reached.

An operator that requires all its input rows ends a pipeline. Later operators wait until those rows are available.

Query operations

Filters

A model filter can ask several TRUE or FALSE questions about each document. Quail processes the document and the first question together. A document that fails a question stops there. A document that passes continues to the next question.

After each answer, Quail removes the question's KV and keeps the document's KV. Removing the added KV while keeping the document's KV is called KV rewind. The next question reuses the document's KV.

A passing document can reach a later operator in the same pipeline without waiting for the filter to finish over the rest of the table. See KV for how Quail stores and reuses KV.

Joins

A model join evaluates a question about a pair of documents. The planner chooses one side as the anchor, whose document KV remains available while Quail evaluates documents from the other side. A document from the other side is a partner.

For example, imagine joining a reviews table with an aspects table. Each aspect describes a subject such as acting or plot. The join asks whether each review discusses each aspect. If reviews are the anchors, Quail processes a review and the fixed join instructions once, then reuses that KV when evaluating its aspects. The result contains review and aspect pairs for which the model answers TRUE.

Several partners can be evaluated in the same batch. Each partner uses the anchor's KV and its own tokens. Partners do not use one another's tokens.

An anchor that passes a filter in the same pipeline can enter the join with its document KV already available. A later join can also reuse that KV if the plan keeps it. See KV retention.

Classification

Classification assigns one of the supplied labels to each input row. It can run in a pipeline and reuse document KV from preceding operators. The selected scoring method determines whether classification needs one or several forward passes.

Label scoring

Label scoring determines how the model selects one of the supplied classification labels. For Qwen3 generative models, the optimizer selects a method using estimated GPU time. The methods can return different labels because they score the choices differently.

For example, imagine classifying customer messages as refund or praise. The generative models can select between these labels in three ways:

A trie is a tree of token sequences in which common starting tokens are stored together. trie_tree evaluates every label sequence. trie_decode follows one sequence, so it can choose a different label from the one with the highest probability for its complete sequence.

For example, suppose refund has two tokens. The first has probability 0.8, and the second has probability 0.6 after the first. The sequence probability is 0.48. If the single token label praise has probability 0.3, trie_tree selects refund.

Additional requirements apply:

• trie_decode requires a model without extra answer positions and a classification that does not reuse document KV from an earlier operator.
• Requesting probabilities excludes trie_decode, because it does not score every label.
• Classifications of joined pairs use letters for generative models.
• DiffusionGemma uses letters with its extra answer positions. See DiffusionGemma for configuration.

Decision 2.0 Kai uses decision_choice. It scores each supplied label as an option and selects the option with the highest score. The model processes all options in one request. It can also return label probabilities for classifications of individual documents.

Classifying joined rows

Classification can also read both documents of a joined pair. It runs in the join's pipeline, on one GPU, and classifies only pairs that the join kept. It reuses the anchor's KV when processing each pair.

GPU execution

Attention paths

During attention, the model combines information from the current token and earlier tokens. A prefix is the token sequence at the start of a request. Quail supports two attention paths for filters and joins:

• Unified attention processes a request's saved prefix and new tokens in one attention call.
• Tree attention processes shared prefixes in one call and each request's own tokens in another. Quail combines the two results.

Tree attention groups requests that use the same saved KV. Grouping can reduce memory reads, but the additional call and combination take time. The planner compares those costs when choosing a path.

Tree attention supports BF16 and FP8 weights. Quail enables it for models with at least two billion parameters, excluding embeddings and output layers. Smaller models use unified attention. DiffusionGemma also uses unified attention because it requires extra answer positions. See Supported models.

Multiple GPUs

Each GPU holds a complete model copy and its own KV. A coordinator is the process that divides the work among GPU workers and collects their results.

• For filters, it assigns documents using their token lengths to balance work across GPUs.
• For joins, anchors stay on the GPU that filtered them so their KV can be reused. Each GPU receives the partner documents needed for its anchors.
• Between groups of joins, it combines the results and distributes the next group's anchors.

Quail supports up to eight GPUs on one host. Each GPU holds all model weights; the weights are not divided between GPUs.