Why does RAG fail on "how many" or aggregate questions across a whole document collection?

Retrieval returns a few chunks, but a count needs a value from every document. ZenDB answers SQL queries, including aggregations, by locating the right section in each document through its template. On three real document collections, it beat RAG baselines by up to 61% in precision and 80% in recall, at slightly higher cost.

Sources: Querying Templatized Document Collections with Large Language Models (ICDE 2025)

Guide: How to Process Documents with LLMs Cheaply

More questions on this topic

All questions about using AI for data and knowledge work