Should I put many records in one prompt so the LLM sees context when classifying?

It can save cost, but it didn't improve accuracy in our tests, because long inputs hurt LLM quality. In HoldUp, batching records into one prompt averaged 43% classification accuracy, compared with 66% one row at a time, and asking the LLM to write a rubric from a sample reached 59%. HoldUp instead gives the LLM small random samples many times, and reached 80%.

Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)

Guide: How to Process Documents with LLMs Cheaply

More questions on this topic

All questions about using AI for data and knowledge work