Supported models and GPUs

We are actively adding support for more models and hardware. This page will be updated as new configurations become available.

Models

The Qwen3 FP8 models use the Quail execution engine, including DeepGEMM for matrix multiplications and FlashAttention 3 for attention on H100.

DiffusionGemma

DiffusionGemma supports filter and join predicates and AI.SCORE. Quail uses the RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic checkpoint and reads one TRUE or FALSE answer without generating a thinking response.

DiffusionGemma can improve accuracy at the cost of longer query times. Check both models on a labeled sample because the result depends on the predicate. On IMDB-2 in QUAIL-B, DiffusionGemma matched 88.89% of the Qwen3 32B reference answers, compared with 76.41% for Qwen3 4B. The query took 32.41 seconds with DiffusionGemma and 21.20 seconds with Qwen3 4B on one H100. Each model used its own prompt format. See the full DiffusionGemma report and Qwen3 4B report for the results and test setup.

Select DiffusionGemma with the model setting:

config = quail.EngineConfig(
    model="diffusion-gemma-26b-a4b-fp8",
    device="h100-sxm",
)

Rerankers

The reranker models use the same Quail executor for AI.SCORE. The generative models also run AI.SCORE; see AI.SCORE for how their score is read. Weights, activations, and KV use BF16. The final yes and no projection and score normalization use FP32. Each input produces one score between 0 and 1 without generating tokens.

Quail tokenizes each document once and the fixed prompt text separately, then joins the token lists. Where the prompt text meets a document, the tokens can differ from tokenizing the joined string, so a score can differ slightly from the reference implementation. Scores near a threshold can select different rows.

GPUs

The device setting must match the hardware where the model runs. Quail uses the device's memory, bandwidth, and FLOP specs to plan the query.

GPU count

Quail supports 1, 2, 4, or 8 GPUs on one host. Each GPU holds one complete model copy. Quail does not split one model across GPUs.

quail.Session(config=quail.EngineConfig(
    model="qwen3-4b-fp8",
    device="h100-sxm",
    gpus=4,
))

Quail partitions the work across GPUs so each GPU processes a different subset of the documents. Results are combined after each operator.

No local GPU? Use Modal

If you don't have a CUDA GPU, you can run Quail inside a Modal GPU function. Quail does not depend on Modal. You write the Modal function, and Quail runs inside it.

Install Modal on the machine you submit from:

uv pip install modal==1.5.4
modal setup

The image below installs Quail 0.1.0 from PyPI. Quail declares its Python dependencies, including vLLM, PyArrow, and its tokenizer. You do not need to install those packages separately. CUDA comes from the base image.

Save this example as run_quail.py:

import modal

app = modal.App("quail-engine")
image = (
    modal.Image.from_registry(
        "nvidia/cuda:13.0.1-devel-ubuntu24.04", add_python="3.12")
    .entrypoint([])
    .uv_pip_install("quail-engine==0.1.0")
)

@app.function(image=image, gpu="H100!", memory=32768, timeout=1200)
def run_query():
    import pyarrow as pa
    import quail

    reviews = pa.table({
        "id": ["r1", "r2"],
        "body": ["The acting was excellent.", "The plot was confusing."],
    })
    config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")
    with quail.Session(config=config) as session:
        session.register("reviews", quail.DocumentProvider.from_table(
            reviews, id_col="id"))
        result = session.sql("""
            SELECT r.id
            FROM reviews r
            WHERE AI.IF(
                PROMPT('Does this review praise the acting? {0}', r.body))
        """, dialect="bq").run()
        return result.collect(), result.report

The imports inside run_query use the packages installed in the GPU image. PyArrow is used here to create the sample input table. Quail loads vLLM and its other execution dependencies internally.

The session, data registration, and execution all happen inside the GPU function. For large datasets, pass a path and open it there instead of sending an Arrow table.

Cache volumes

Mount Modal volumes to persist model weights and compiled kernels across runs:

weights = modal.Volume.from_name("quail-hf-cache", create_if_missing=True)
kernels = modal.Volume.from_name("quail-kernel-cache", create_if_missing=True)

image = image.env({
    "HF_HOME": "/model-cache",
    "QUAIL_CACHE_DIR": "/kernel-cache/quail",
})

@app.function(
    image=image, gpu="H100!", memory=98304, timeout=1200,
    volumes={"/model-cache": weights, "/kernel-cache": kernels},
)
def run_query():
    # Use the same run_query body from above.
    ...

Quail's own cache is controlled by QUAIL_CACHE_DIR (default ~/.cache/quail/kernels). Other libraries use their own environment variables: HF_HOME, DG_CACHE_DIR, TRITON_CACHE_DIR, VLLM_CACHE_ROOT.

Submit and retrieve

@app.local_entrypoint()
def main():
    call = run_query.spawn()
    print(f"function call id: {call.object_id}", flush=True)
    rows, report = call.get()
    print(rows.to_pylist())
    print(report)

Run the file from the machine where you installed Modal:

modal run --detach run_quail.py 2>&1 | tee run_quail.log

Keep the function call id. After a disconnect, retrieve the result with modal.FunctionCall.from_id("fc-...").get(). For long runs, use modal run --detach.

The function should return the results before the session closes:

return result.collect(), result.report

If you want to save results to a Modal volume, write them to the mounted path and call volume.commit() inside the function. Quail does not do this automatically.

Adding a model or device

See Extending Quail for how to register a new model or device.