How do I get consistent 1-10 scores from an LLM across many items?

Scoring each item alone on an absolute scale drifts; comparing items gives steadier results. In HoldUp, we group similar records, ask the LLM which group should score higher, and sort the groups with an integer linear program. On 5 scoring datasets, row-by-row scoring averaged 48% accuracy and HoldUp 64%.

Sources: Semantic Data Processing with Holistic Data Understanding (Preprint)

Guide: How to Process Documents with LLMs Cheaply

More questions on this topic

All questions about using AI for data and knowledge work