Should an agent use regex or an LLM to pull values like dates out of free-text columns?
Regex works for a few fixed formats and breaks on varied natural language. In DAB, all three agents we tested defaulted to regex for text extraction. On the patents dataset, dates appear as "dated 5th March 2019" and "March the 18th, 2019"; no agent's regex covered them, and every agent scored 0%. A regex for MALE also matched inside FEMALE.
Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)
More questions on this topic
- Why are AI functions in SQL (Snowflake Cortex, BigQuery AI, Databricks ai_query) so slow and expensive on big tables?
- How do I extract information accurately from thousands of long documents with an LLM?
- My LLM misses items when I ask it to find every instance of something in a long document. Should I split the prompt or the document?
- Should I ask an LLM to extract dozens of fields from a long document in one prompt, or split it up?
- How do I optimize a multi-step LLM pipeline for both accuracy and cost, not just one prompt at a time?
- Can I use a cheaper LLM for most rows without losing accuracy?
- Is there something better than a model cascade for LLM classification over documents?
- How do I match records across two tables with an LLM without paying for every pair?
- In what order should I run several LLM filters on the same data?
- Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?
- Should I put many records in one prompt so the LLM sees context when classifying?
- How do I get consistent 1-10 scores from an LLM across many items?
- How do I run SQL queries over thousands of similar PDFs, such as reports or forms that follow one template?
- Why does RAG fail on "how many" or aggregate questions across a whole document collection?
- How do I deduplicate records with an LLM without missing too many true matches?
- How do I add a summary of each user's history and feedback to my prompts without slowing down responses?
- Can LLM agents reliably answer business questions over real company databases, not just text-to-SQL benchmarks?
- Why do text-to-SQL and data agents return wrong answers? Is it picking the wrong table?
- Do AI agents need a different kind of database than human analysts do?
- Why does my LLM agent run so many SQL queries for one question, and how do I cut them down?
- How do I write assertions or guardrails that catch bad LLM outputs in production?
- Can an LLM generate the test criteria or assertions for my prompt?
- How do I catch bad data before it gets into a retraining run without drowning in false alerts?
- How do I monitor an ML model in production when labels arrive late or not at all?
- My model's accuracy on a rolling window dropped. Is the model actually getting worse?
- Can I use Prometheus and Grafana to monitor ML model accuracy?
- Should I fine-tune a model or use RAG to give an LLM my company's knowledge?
- Do long-context models make RAG obsolete?
- How should I evaluate a RAG system in production?