How do I deduplicate records with an LLM without missing too many true matches?

Asking an LLM about each pair alone tends to give high precision and low recall; using the other answers to fix mistakes helps. In Revisiting Prompt Engineering via Declarative Crowdsourcing (CIDR 2024), gpt-3.5-turbo matched citation pairs with 0.952 precision but 0.503 recall. Marking two records as duplicates when the LLM linked them through other records raised F1 by over 6%.

Sources: Revisiting Prompt Engineering via Declarative Crowdsourcing (CIDR 2024)

Guide: How to Process Documents with LLMs Cheaply

More questions on this topic

All questions about using AI for data and knowledge work