Do long-context models make RAG obsolete?

No. Even with a 10M-token window you still have to choose what goes in it. In What We've Learned From a Year of Building with LLMs (O'Reilly Radar), we note there was little evidence that models reason well over very long inputs beyond needle-in-a-haystack tests. Irrelevant context can distract the model, and inference time grows with context length, so reading everything for every question costs more.

Sources: What We've Learned From a Year of Building with LLMs (O'Reilly Radar)

More questions on this topic

All questions about using AI for data and knowledge work