Supported models and GPUs

Set model to the model you want to run and device to your GPU type.

Models

FP8 is an 8 bit floating point format. BF16 is a 16 bit floating point format. All listed models store KV in BF16.

Caveats for d1-2B:

• No AI.MAP.
• AI.SCORE over one table only.
• Joins cost more tokens than on Qwen3.
• Answers can shift when questions share a pass.
• "d1-2b-fp8" is quantized by Quail at load, not by Liquid; check its answers.

AI.IF supports filters and joins. AI.SCORE returns a score, and AI.CLASSIFY assigns a label. AI.EXTRACT copies an answer from a document, and AI.MAP writes one. AI.SAE_FEATURES returns the sparse autoencoder features of a prompt. See the SQL reference for examples.

GPUs

L40S does not support DiffusionGemma, Qwen3.5, or Qwen3.8.

Configure a session

For example, to run Qwen3 4B on one H100 SXM:

import quail

config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")

Pass config to quail.Session. Quail downloads the model weights when they are not already available.

Use multiple GPUs

Set gpus to 1, 2, 4, or 8 GPUs on one machine. The default is one. Each GPU holds a complete model copy and processes part of the input. Quail does not split a model's weights across GPUs.

config = quail.EngineConfig(
    model="qwen3-4b-fp8",
    device="h100-sxm",
    gpus=4,
)

DiffusionGemma

DiffusionGemma uses the RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic checkpoint. Set HF_TOKEN before running it because access to the weights is gated.

For AI.CLASSIFY and AI.SCORE, canvas_draws limits the number of attempts per document. The default is four. Quail can make additional attempts when the first answer's probabilities are spread across labels. It averages the probabilities from those attempts.

Set canvas_draws=1 to use one attempt per document:

config = quail.EngineConfig(
    model="diffusion-gemma-26b-a4b-fp8",
    device="h100-sxm",
    canvas_draws=1,
)

Classification of joined rows always uses one attempt. The setting does not change AI.IF execution.

Decision 2.0 Kai 0.6B

Decision 2.0 Kai 0.6B is a model from the vLLM Semantic Router team. It is fine tuned from Qwen3 0.6B to score supplied answers and return their probabilities in one forward pass.

Quail uses the model for these functions:

• AI.IF selects between "Yes" and "No" for a filter or join condition.
• AI.SCORE returns the probability of "Yes" for that condition.
• AI.CLASSIFY selects the supplied label with the highest score.

config = quail.EngineConfig(
    model="decision-2.0-kai-0.6b-bf16",
    device="h100-sxm",
)

Use the existing SQL query syntax. Quail prepares the prompt for the model.

Calibration describes how closely predicted probabilities match observed answer frequencies. Check calibration on labeled examples from your task before choosing a score threshold.

See the vLLM Semantic Router team's blog post introducing decision models for the approach. The post covers Decision 1.0; the Kai 2.0 model card describes the model supported here.

Qwen3.5 2B Base with Qwen-Scope SAE

The qwen3.5-2b-sae-bf16 model serves AI.SAE_FEATURES only. Quail loads layers 0 to 11 of the 24 layers of Qwen3.5-2B-Base. The sparse autoencoder (SAE) is the Qwen-Scope Qwen/SAE-Res-Qwen3.5-2B-Base-W32K-L0_100 SAE for layer 11. The SAE reads the residual stream after layer 11. It has 32,768 features and keeps the 100 strongest features of each token.

The model has no layers after layer 11, so it cannot answer AI.IF, AI.SCORE, AI.CLASSIFY, or AI.EXTRACT. Use another model in a separate session for these functions.

config = quail.EngineConfig(
    model="qwen3.5-2b-sae-bf16",
    device="h100-sxm",
)

No local GPU? Use Modal

You can run Quail in a Modal GPU function. Install Modal and authenticate on the machine that submits the function:

uv pip install modal==1.5.4
modal setup

For example, the program below selects movie reviews that praise the acting. Save it as run_quail.py. It runs on one H100 and saves model weights and compiled kernels on Modal volumes for later runs.

import modal

app = modal.App("quail-engine")
weights = modal.Volume.from_name("quail-hf-cache", create_if_missing=True)
kernels = modal.Volume.from_name("quail-kernel-cache", create_if_missing=True)
image = (
    modal.Image.from_registry(
        "nvidia/cuda:13.0.1-devel-ubuntu24.04", add_python="3.12")
    .entrypoint([])
    .uv_pip_install("quail-engine==0.1.0")
    .env({
        "HF_HOME": "/model-cache",
        "QUAIL_CACHE_DIR": "/kernel-cache/quail",
    })
)

@app.function(
    image=image,
    gpu="H100!",
    memory=98304,
    timeout=1200,
    volumes={"/model-cache": weights, "/kernel-cache": kernels},
)
def run_query():
    import pyarrow as pa
    import quail

    reviews = pa.table({
        "id": ["r1", "r2"],
        "body": ["The acting was excellent.", "The plot was confusing."],
    })
    config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")
    with quail.Session(config=config) as session:
        session.register("reviews", quail.DocumentProvider.from_table(
            reviews, id_col="id"))
        result = session.sql("""
            SELECT r.id
            FROM reviews r
            WHERE AI.IF(
                PROMPT('Does this review praise the acting? {0}', r.body))
        """, dialect="bq").run()
        return result.collect(), result.report

@app.local_entrypoint()
def main():
    call = run_query.spawn()
    print(f"function call id: {call.object_id}", flush=True)
    rows, report = call.get()
    print(rows.to_pylist())
    print(report)

The example installs Quail 0.1.0 in the GPU image. Collect the rows inside the session before returning from the function.

Run and retrieve the result

modal run --detach run_quail.py 2>&1 | tee run_quail.log

Keep the function call ID printed in the log. If you disconnect, retrieve the result with that ID:

rows, report = modal.FunctionCall.from_id("fc-...").get()

The example returns rows and a report. To save them on a Modal volume, write them to its mounted path and call volume.commit() in the function.

Add a model or device

See Custom models and GPUs to register another model or device.