Why do text-to-SQL and data agents return wrong answers? Is it picking the wrong table?

Usually not; agents mostly find the right data and then compute the wrong thing. In DAB, we labeled failed agent runs: wrong implementations, such as a regex that matched part of an ISBN instead of a year, caused 57% of wrong answers. Wrong plans caused 31%, and picking the wrong data only 11%.

Sources: Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (Preprint)

More questions on this topic

All questions about using AI for data and knowledge work