The race to expand context windows has produced some genuinely impressive numbers. Models from Google, Anthropic, and OpenAI now support anywhere from 200,000 to over a million tokens in a single prompt. That’s roughly the equivalent of several full-length novels fed to a model in one shot. The engineering required to make that work at inference speed is not trivial.
But there’s a gap between what’s technically possible and what’s actually happening in daily use, and it’s wider than the marketing suggests.
The Retrieval Problem Didn’t Disappear
Long context doesn’t mean uniform attention across that context. Research from multiple academic groups has consistently shown that transformer-based models tend to weight tokens near the beginning and end of a prompt more heavily than those buried in the middle - a phenomenon sometimes called the “lost in the middle” problem. Anthropic, Google, and others have made efforts to address this, with varying claims about improvement, but the underlying architectural tendency hasn’t been fully solved.
What this means practically: stuffing 800 pages of documentation into a context window and asking a model to answer questions about page 400 is not the same as the model having actually read and retained page 400. The token is present. Whether it meaningfully influences the output is a different question.
Enterprise Adoption Is Still Retrieval-Augmented
Despite the availability of massive context windows, most serious enterprise AI deployments still use retrieval-augmented generation. They chunk documents, embed them, pull relevant chunks at query time, and feed a relatively small, targeted context to the model. This approach remains more predictable and cheaper. Long context pricing, even as rates have dropped, adds up at scale.
The tooling built around RAG - vector databases, chunking strategies, embedding pipelines - represents real investment that enterprises aren’t going to abandon just because a model can accept more tokens.
What Long Context Actually Solves
It’s genuinely useful for certain tasks: analyzing a single large codebase, comparing a full contract against a set of criteria, keeping a long conversation coherent without summarization artifacts. These are real workflows where long context earns its keep.
But that’s a narrower use case than the capability implies. The dominant mental model sold alongside these announcements - that you can simply feed a model everything and it will reason over all of it reliably - remains more aspirational than operational. The window is open. What the model does with the view is still the hard part.