Explore topics in support conversations

Quail can help discover recurring topics in text without a predefined list of labels.

In this tutorial, we use 1,320 support conversations from τ-bench to explore topics in airline bookings and retail orders. We'll look for common types of requests and retrieve examples of each.

Sparse feature vectors

AI.SAE_FEATURES represents each request as a sparse feature vector. Most entries are zero. Each request activates a small set of features, and a larger value means a stronger activation. Feature IDs are shared across requests, but they have no topic labels attached to them.

For example, an illustrative vector {17: 0.8, 24933: 2.1} activates features 17 and 24933. All other entries are zero.

Dense embeddings are useful for comparing documents, but have limitations for exploratory analysis:

1. Embeddings from a small encoder model, such as all-MiniLM-L6-v2, may not capture all the topics and details in a document.
2. Individual features are difficult to interpret. A feature does not usually correspond to a topic or a property of the document.

Sparse features offer a chance to inspect individual topics. Requests that strongly activate one feature may share a recognizable topic. Research on interpretable sparse features provides the basis for this approach. Sparse features can miss details too.

Get features for each request

The conversations are loaded from the QUAIL-B support_traces table. Each row contains one conversation's id, its opening request, and a task_id. Several conversations can come from the same customer task.

The query asks what the customer wants. It runs on one H100 with the qwen3.5-2b-sae-bf16 model. The complete script and its Modal command are at the end of this page.

import quail
from quail_b import load_table

conversations = load_table("support_traces", scale_factor=1.0)
config = quail.EngineConfig(model="qwen3.5-2b-sae-bf16", device="h100-sxm")

with quail.Session(config) as session:
    session.register("conversations", quail.DocumentProvider.from_table(
        conversations, id_col="id"))
    features = session.sql(r"""
        SELECT t.id, intent.feature, intent.strength
        FROM conversations t,
             UNNEST(AI.SAE_FEATURES(
                 PROMPT('{0}\n\nWhat is the customer asking for?', t.request),
                 pooling => 'last')) AS intent
    """).collect()

AI.SAE_FEATURES returns up to 100 positive feature values per request with this configuration. UNNEST(...) AS intent expands each list into rows:

Find common features and inspect their requests

The demo reduces the weight of features that occur in nearly every request, then keeps three features per request. The analysis counts how many requests keep each feature and selects three representative requests from different customer tasks.

from demos.support_intents import analyze

report = analyze(conversations, features)
for feature in report["intents"]:
    print(feature["feature"], feature["requests"], feature["examples"])

The table shows four features from the demo run. Each example is the first sentence of the request with the largest weighted activation. The possible topics come from inspecting all three examples per feature. The request counts are after the three-feature selection.

Feature 24933 suggests a flight cancellation topic. Feature 4561 combines several topics, so it needs more inspection. A feature's meaning depends on the requests that activate it.

Retrieve more requests for a topic

The feature table can retrieve more examples without another model run. The following code selects feature 24933, sorts by activation strength, and looks up the original requests by conversation ID:

import pyarrow.compute as pc

matches = features.filter(pc.equal(features["intent.feature"], 24933))
matches = matches.sort_by([("intent.strength", "descending")])
positions = pc.index_in(matches["t.id"], value_set=conversations["id"])
requests = conversations.take(positions).select(["id", "request"])
print(requests.slice(0, 10).to_pylist())

Run the demo

The complete analysis is in demos/support_intents.py. The Modal wrapper loads the data and runs the query on one H100:

uv run modal run --detach demos/support_intents_modal.py \
    2>&1 | tee /tmp/quail-support-intents.log

The wrapper prints its Modal function call ID. It saves the feature table as features.parquet and the counts and example requests as report.json under demos/support-intents/ on the quail-results volume.