Custom models and GPUs

Every planning decision in Quail comes from a ModelSpec and a DeviceSpec. Adding support for a new model or GPU means registering a spec.

ModelSpec

A ModelSpec describes the model architecture. Every field except params and w_mem_bytes comes from the checkpoint's config.json.

from quail.specs import ModelSpec

MY_MODEL = ModelSpec(
    name="qwen3-4b-fp8",
    params=4.0e9,
    layers=36,
    hidden=2560,
    n_q=32,
    n_kv=8,
    d_head=128,
    ffn_width=19456,
    w_bytes=1.0,
    hf_name="Qwen/Qwen3-4B-FP8",
    revision="96b30dc...",
    vocab=151936,
    tied_head=True,
)

Source: quail/specs/base.py.

revision pins the Hugging Face hub commit so the worker resolves weights from cache without network round trips.

The planner uses derived properties: kappa (KV bytes per token), W_resident (weight bytes on GPU after discarding the full output head), and act_per_token (activation memory per token).

Which models work with a spec alone

The Quail executor's GPU kernels are written for the Qwen3 architecture (QK-norm, gated SiLU MLP, FP8 block-quantized weights). Any Qwen3-family FP8 checkpoint can be added by registering a ModelSpec with the correct dimensions.

A model with a different architecture (different attention layout, different activation function, different weight format) would need new GPU kernels in quail/backends/quail/executor/, or could run through a custom inference engine like vLLM or SGLang, which handle model loading independently.

DeviceSpec

A DeviceSpec describes the GPU hardware.

from quail.specs import DeviceSpec

MY_DEVICE = DeviceSpec(
    name="h100-sxm",
    mem_bytes=80e9,
    hbm_bw=3.35e12,
    peak_flops=1.979e15,
    bf16_flops=0.989e15,
)

Source: quail/specs/base.py.

All values are per GPU. peak_flops is the dense FP8 rate. The H100 datasheet quotes rates with 2:1 structured sparsity; divide by 2 for the dense rate Quail uses. bf16_flops is the dense BF16 rate, used for attention (FlashAttention runs in BF16).

Registration

registry = quail.ExtensionRegistry.with_built_ins()
registry.register_model(MY_MODEL)
registry.register_device(MY_DEVICE)

Then use the registered names in EngineConfig:

config = quail.EngineConfig(
    model="my-model",
    device="my-device",
    gpus=1,
)