Can LLM agents reliably answer business questions over real company databases, not just text-to-SQL benchmarks?
Not yet, once the data is spread across systems and messy. Our DAB benchmark has 104 questions over 17 datasets in PostgreSQL, MongoDB, SQLite, and DuckDB, based on a study of six enterprises building data agents. The best agent, GPT-5.5 in Codex, answered 57% correctly on the first try, and 70% within five tries.
Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)
More questions on this topic
- Why are AI functions in SQL (Snowflake Cortex, BigQuery AI, Databricks ai_query) so slow and expensive on big tables?
- How do I extract information accurately from thousands of long documents with an LLM?
- My LLM misses items when I ask it to find every instance of something in a long document. Should I split the prompt or the document?
- Should I ask an LLM to extract dozens of fields from a long document in one prompt, or split it up?
- How do I optimize a multi-step LLM pipeline for both accuracy and cost, not just one prompt at a time?
- Can I use a cheaper LLM for most rows without losing accuracy?
- Is there something better than a model cascade for LLM classification over documents?
- How do I match records across two tables with an LLM without paying for every pair?
- In what order should I run several LLM filters on the same data?
- Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?
- Should I put many records in one prompt so the LLM sees context when classifying?
- How do I get consistent 1-10 scores from an LLM across many items?
- How do I run SQL queries over thousands of similar PDFs, such as reports or forms that follow one template?
- Why does RAG fail on "how many" or aggregate questions across a whole document collection?
- How do I deduplicate records with an LLM without missing too many true matches?
- How do I add a summary of each user's history and feedback to my prompts without slowing down responses?
- Why do text-to-SQL and data agents return wrong answers? Is it picking the wrong table?
- Should an agent use regex or an LLM to pull values like dates out of free-text columns?
- Do AI agents need a different kind of database than human analysts do?
- Why does my LLM agent run so many SQL queries for one question, and how do I cut them down?
- How do I write assertions or guardrails that catch bad LLM outputs in production?
- Can an LLM generate the test criteria or assertions for my prompt?
- How do I catch bad data before it gets into a retraining run without drowning in false alerts?
- How do I monitor an ML model in production when labels arrive late or not at all?
- My model's accuracy on a rolling window dropped. Is the model actually getting worse?
- Can I use Prometheus and Grafana to monitor ML model accuracy?
- Should I fine-tune a model or use RAG to give an LLM my company's knowledge?
- Do long-context models make RAG obsolete?
- How should I evaluate a RAG system in production?