Somewhere along the way, ‘context window’ became a marketing metric. Gemini 1.5 Pro launched with a one-million-token context window to audible gasps. Claude’s 200K window gets cited in product comparisons like it’s a RAM spec. The implicit promise being sold is continuity - that a bigger window means the model knows you better, tracks your preferences, picks up where you left off. None of that is what a context window does.
A context window is a working buffer, not a memory system. Everything inside it is weighted together mathematically at inference time. When the session ends, it’s gone. Nothing persists. The model doesn’t ‘remember’ your last conversation any more than a whiteboard remembers the meeting where someone used it. Retrieval-augmented generation (RAG) systems and explicit memory tools like those OpenAI and others have begun shipping are closer to actual memory - but they’re architecturally separate from the context window, and even they work more like indexed search than recollection.
The conflation matters because it shapes what developers build and what users expect. Products get designed around the assumption that stuffing more into a single session is a substitute for building actual persistence layers. You see this in AI coding tools that expect users to paste entire codebases into a chat rather than maintaining a structured project model. You see it in AI assistants that ask you your preferences every single conversation because no one built the part that stores the answer.
Why Bigger Isn’t the Fix
Large context windows do solve real problems - processing a long document without chunking it awkwardly, for instance, or holding an entire codebase in view for a single refactor. Those are legitimate uses. But they don’t scale as solutions to the continuity problem, partly for technical reasons (attention mechanisms don’t weight distant tokens equally; performance degrades at the edges of very large contexts) and partly for practical ones: most users aren’t pasting million-token documents. They’re having a fifteen-minute work session and wanting the tool to feel less amnesiac.
Memory as a product feature has been slow, fragmented, and inconsistent across the industry. OpenAI’s memory feature, launched in 2024, is opt-in and often arbitrary about what it retains. Most competitors are still treating it as secondary.
That asymmetry - enormous investment in raw context size, comparatively modest investment in persistent user modeling - reflects what’s easier to benchmark rather than what makes the experience better. Window size fits in a spec sheet. The feeling that a tool actually knows what you’re working on doesn’t.