Quail Server

Quail Server runs queries on a GPU machine over HTTP. Start the server, then pass its URL as the endpoint of a Session.

The server plans each query and loads its model. Later queries can reuse the loaded model. Queries continue after the client disconnects, and other clients can retrieve their saved records.

Start the server

Install the server extra on the GPU machine. Then run quail-server.

uv pip install "quail-engine[server]"
quail-server

By default, the server accepts every built-in model, detects GPU 0, stores data in ~/.quail/server, and listens on 127.0.0.1:8642. At startup it prints the device, accepted models, data directory, and endpoint.

Each query selects a model in EngineConfig. The first query loads its weights and compiles its kernels. Queries for the same model reuse it. A query for another model replaces it because the model weights and KV use the available GPU memory.

Quail recognizes an H100 80GB HBM3 as h100-sxm and an RTX PRO 6000 Blackwell as rtx-pro-6000-blackwell-server. If detection fails, pass one of these names with --device.

From another machine

The default address accepts connections only from the server machine. To connect from another machine, listen on every interface and require a bearer token.

QUAIL_SERVER_TOKEN=<token> quail-server --host 0.0.0.0

Use endpoint="http://<host>:8642" on a trusted private network. For an untrusted network, put the server behind an HTTPS reverse proxy and use the proxy URL. Plain HTTP does not encrypt the token or query data.

Set the same QUAIL_SERVER_TOKEN on each client. Session sends it as a bearer token with every request.

Options

Use these options to restrict accepted configurations or change storage and network settings.

The server stores its SQLite database, uploads, and results in --data-dir. Reuse that directory after a restart. Queued queries resume. Queries that were planning or running become interrupted and must be submitted again.

What the machine needs

Quail Server supports an H100 SXM or RTX PRO 6000 Blackwell Server. Each GPU holds one complete model copy. See Supported models and GPUs for valid configurations.

Model weights and compiled kernels use their standard caches. DiffusionGemma also requires HF_TOKEN because its weights are gated.

Connect from Python

Pass the server URL as endpoint. Register tables and run SQL through the resulting session. This example watches the query and reads its saved result.

import pyarrow as pa
import quail

reviews = pa.table({
    "id": ["r1", "r2"],
    "body": ["The ending was excellent.", "I liked the soundtrack."],
})
sql = """
    SELECT r.id
    FROM reviews r
    WHERE AI.IF(PROMPT('Does this discuss the ending? {0}', r.body))
"""
config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")
with quail.Session(config=config, endpoint="http://127.0.0.1:8642") as session:
    session.register(
        "reviews",
        quail.DocumentProvider.from_table(reviews, id_col="id"),
    )
    run = session.sql(sql, dialect="bq").submit()
    for status in run.watch():
        print(status.phase["message"])
    table = run.result().collect()
    print(table)

sql() returns a RemoteQuery. The client uses the Python standard library for HTTP and pyarrow for record batches. Only the server machine needs the server extra.

When authentication is enabled, set QUAIL_SERVER_TOKEN on the client. Session has no token argument. A token passed to ServerClient applies only to that client instance.

Tables on a remote session

register() prepares each table for the server. The server must be able to read it after the client exits. Unsupported providers raise TypeError and are not registered.

The client checks the content hash before uploading a snapshot. Identical tables are uploaded only once. Closing the session removes local staging files, but accepted inputs remain on the server.

There is no deletion or garbage-collection API. An operator must remove uploaded snapshots and saved query artifacts.

Submit without waiting

run() and collect() wait for rows. submit() returns a QueryRun as soon as the server saves the query record. The query continues if the client disconnects. Use session.get_run(query_id) from another process to reconnect to it.

with quail.Session(
    config=config,
    endpoint="http://127.0.0.1:8642",
) as session:
    session.register(
        "reviews",
        quail.DocumentProvider.from_table(reviews, id_col="id"),
    )
    run = session.sql(sql, dialect="bq").submit(
        query_id="nightly-2026-09-21", timeout_s=1000)

with quail.Session(
    config=config,
    endpoint="http://127.0.0.1:8642",
) as session:
    run = session.get_run("nightly-2026-09-21")
    for status in run.watch():
        print(status.state, status.progress)
    result = run.result()

watch() yields a QueryStatus each time the saved record changes. cancel() removes a queued query. Cancelling a running query stops the model process and leaves the record in cancelled. The next query loads its model again.

Use a stable query_id to retry a submission. If a response is lost, submit the same query with the same id. The server returns the existing run. If the query, engine config, inputs, or timeout differ, the server returns HTTP 409 and QueryIdConflictError.

When omitted, query_id is random. An id has 1 to 128 characters. It must start with a letter or digit. The remaining characters may also include ., _, :, and -.

session.session_id identifies the client session. The server saves it on each submitted record. List those records with GET /v1/queries?session_id=<id>.

A record moves from queued to planning to running. It ends as succeeded, failed, interrupted, or cancelled. The phase field reports more detail, including input resolution, model loading, and execution. These are saved state changes, not time estimates. A terminal record does not change again.

The server records succeeded only after reopening the saved result, report, and answer tables and checking the result row count.

Set timeout_s to limit execution. Otherwise the server uses --default-timeout. The timer starts when the query leaves the queue. A timeout stops the model process and records a TimeoutError. Values above --max-timeout are rejected at submission.

RemoteQuery

session.sql on a server session returns a RemoteQuery. It holds the statement text. The server compiles and plans that text when the query runs. A local session returns a Query instead. After you submit, the plan text is on the record at QueryStatus.plan["text"].

Attributes

Methods

submit

Send the query to the server and return once the record is saved. The query keeps running if the client disconnects. Submitting the same query_id with the same statement, engine config, inputs, and timeout returns the existing run. The same id with a different submission is HTTP 409, QueryIdConflictError.

Parameters

Returns

QueryRun.

run

Submit the query, wait until the saved result is ready, and return it. A record that ended failed, interrupted, or cancelled raises QueryFailedError. The exception carries the saved status. Remote queries always use the server's plan. Passing a non-None plan raises RuntimeError.

Parameters

Returns

QueryResult.

collect

Submit the query, wait, and return one Arrow table.

Parameters

Returns

pyarrow.Table.

execute_stream

Submit the query, wait, and stream the rows.

Parameters

Returns

pyarrow.RecordBatchReader.

QueryRun

A handle to one record the server has accepted. submit returns one. session.get_run(query_id) returns a handle to a record that is already there.

Attributes

Methods

status

Read the current saved record.

Returns

QueryStatus.

watch

Yield the current record, then each newer revision, until the query is finished. The default poll is 30 seconds, so a progress update that was replaced before the next poll is skipped. A lost connection raises ServerError. Calling watch again continues from the record as it stands then.

Parameters

Returns

An iterator of QueryStatus.

wait

Wait until the record is finished and return its final QueryStatus. A record that ended failed, interrupted, or cancelled raises QueryFailedError. The exception has the final snapshot on its status attribute.

Parameters

Returns

The successful final QueryStatus.

result

Wait until the record is finished and return the saved rows. A record that ended failed, interrupted, or cancelled raises QueryFailedError, and the exception carries the saved status.

Parameters

Returns

QueryResult, built from the saved Arrow files and the report.

stream_result

Wait until the record is finished, then stream the rows. The failure behavior matches result.

Parameters

Returns

pyarrow.RecordBatchReader. The server chooses the batch size.

reader = run.stream_result()
try:
    for batch in reader:
        print(batch)
finally:
    reader.close()

cancel

Ask the server to stop this query. A queued query ends immediately. A query that is already running has its model process killed, and the record ends cancelled. The next query loads the model again.

Returns

QueryStatus. The snapshot after the cancel request.

answers

Return answer entries saved so far, starting at entry after. The entry shapes are in Answers while the query runs.

Parameters

Returns

A dict with answers (the entries), next (the index to pass as after on the next call), and done (whether more entries can still arrive).

stream_answers

Yield each saved answer entry in order. The iterator returns when the query is finished and every saved entry has been yielded.

Parameters

Returns

An iterator of answer dicts.

QueryStatus

One saved snapshot of a query record. status, watch, and GET /v1/queries/{id} return this object. Every write replaces the snapshot and increases revision. A record in succeeded, failed, interrupted, or cancelled stays as it is. The server writes succeeded only after it has opened the saved result, the report, and the answer tables again and checked the result's row count.

Attributes

phase

spec

config

inputs

Each value is one registered table.

A snapshot has kind "snapshot", content_id (the SHA-256 of the Arrow file), id_col, and columns.

A Hugging Face dataset has kind "hf", dataset, config, split, revision, and id_col.

progress

plan

error

result

Answers while the query runs

The server saves answers while a query runs. Clients can display them before the final result is ready.

A filter entry contains one completed chunk of documents. Each item gives the row index, last stage asked, and whether the document passed. A join entry arrives after an anchor document finishes. matches contains the partner rows that returned true. asked gives the number of evaluated pairs.

An AI.SCORE entry contains one batch of aligned rows and scores. An evict entry identifies a document prefix removed from KV by alias, row index, and token length.

for entry in run.stream_answers():
    if entry["kind"] == "filter":
        survivors = [row for row, _stage, passed in entry["documents"] if passed]
    elif entry["kind"] == "join":
        print(entry["anchor"], entry["document"],
              len(entry["matches"]), "of", entry["asked"])
    elif entry["kind"] == "score":
        ...
    elif entry["kind"] == "evict":
        print(entry["alias"], entry["document"], entry["tokens"])

stream_answers() yields entries in order until the query and stream are complete. answers(after=N) returns entries starting at N, the next index, and whether more entries may arrive. The saved count is status().progress["answers_saved"].

Read live entries through these methods or the answers route. The internal answers.jsonl file is not available through /files/{name}.

After a successful run, the server writes final Arrow answer tables. They contain row indexes, predicate metadata, and boolean answers. They do not contain source document text. The final result contains only columns selected by the SQL query.

Reading the plan

The server compiles and plans submitted SQL. When status.plan is not None, read the plan from run.status().plan["text"]. Remote queries always use the server's plan.

Local Query objects support planning before execution. The Python builder is also available only on a local session.

In the browser

Open http://<host>:8642/queries/<query_id> to inspect a query. The page shows its state, progress, plan, errors, SQL, inputs, and the first 20 result rows. It updates through server-sent events while the query runs. Open http://<host>:8642/ to list recent queries.

If the server requires a token, open the page once with #token=<token> appended to the URL. The page saves the token in browser local storage and removes it from the address bar. It asks for a token after an unauthorized response if none is saved.

The page sends the token when it fetches result and report files. It then saves those files locally.

HTTP API

The Python client and status page use these routes. When authentication is enabled, every /v1 route requires Authorization: Bearer <token>. The HTML pages load without this header and attach it to their API requests.

A submission body has sql, dialect, order, config, inputs, query_id, session_id, and timeout_s. dialect is bq or snowflake, the same values as session.sql.

{
  "sql": "SELECT r.id FROM reviews r WHERE AI.IF(PROMPT('Positive? {0}', r.body))",
  "dialect": "bq",
  "order": null,
  "config": {
    "model": "qwen3-4b-fp8",
    "device": "h100-sxm",
    "gpus": 1,
    "backend": "quail"
  },
  "inputs": {
    "reviews": {
      "kind": "snapshot",
      "content_id": "<lowercase sha256 of the uploaded Arrow IPC file>",
      "id_col": "id"
    }
  },
  "query_id": "reviews-2026-09-22",
  "session_id": "raw-http-client",
  "timeout_s": 1000
}

Upload each Arrow IPC input with PUT /v1/inputs/{content_id} before submitting the request. A Hugging Face input uses kind, dataset, config, split, revision, and id_col instead.

On Modal

Deploy quail.server.modal_app to run Quail Server on one H100 in the existing quail-engine Modal app.

The server extra does not install Modal. To deploy, use a repository checkout with an installed and authenticated Modal CLI. The image uses the checkout's uv.lock, development dependencies, and local quail package.

modal secret create quail-server-token QUAIL_SERVER_TOKEN=<token>
modal deploy -m quail.server.modal_app

modal deploy prints the endpoint URL. Pass it as endpoint. The URL is public, so create the quail-server-token secret before deployment. Clients must set the same QUAIL_SERVER_TOKEN.

The deployment accepts these built-in models: qwen3-4b-fp8, qwen3-32b-fp8, diffusion-gemma-26b-a4b-fp8, qwen3-reranker-0.6b-bf16, and qwen3-reranker-4b-bf16.

For DiffusionGemma, add HF_TOKEN to the same secret:

modal secret create quail-server-token \
  QUAIL_SERVER_TOKEN=<server-token> HF_TOKEN=<hugging-face-token>

Without HF_TOKEN, DiffusionGemma cannot download its weights.

Results and inputs go to the quail-results Volume under /results/quail-server/. The live SQLite database stays on the container's local disk at /tmp/quail-server/quail.sqlite3. A checkpoint thread copies it to the Volume with SQLite's online backup and commits the Volume.

The server commits a checkpoint before returning a new query id. It then attempts to checkpoint durable changes every second and progress-only changes at most every 30 seconds. Failed checkpoints are logged and retried. A restart restores the latest successful checkpoint, which can be older than the latest state a client observed.

New containers restore the database before serving requests. max_containers=1 keeps one writer. A container stops after 15 idle minutes. The next request starts a new container, and its first query loads the model again.