Why is my LLM classifier inaccurate when I label records one at a time, even with a good prompt?

Often the right label depends on the rest of the dataset, and a model that sees one row can't know that. For example, a ticket can be urgent next to minor issues and routine next to security bugs. In HoldUp, labeling rows one at a time averaged under 70% accuracy on 10 classification datasets. HoldUp clusters related records with the LLM and labels the clusters jointly, reaching 80% on average and up to 33% higher.

Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)

Guide: How to Process Documents with LLMs Cheaply

More questions on this topic

All questions about using AI for data and knowledge work