The benchmark scores keep climbing. Context windows are now measured in millions of tokens. Response quality, on most objective measures, is meaningfully better than it was two years ago. And yet there’s a pattern emerging across the major AI assistants - GPT-4o, Gemini 1.5, Claude - that the improvement metrics don’t capture: they have become extraordinarily reluctant to admit the limits of their own knowledge.
This isn’t about hallucination in the traditional sense. The well-documented problem of models fabricating citations or inventing names is real, but most labs have made genuine progress on the most egregious cases. The subtler issue is confident vagueness - where a model produces a fluent, reassuring, technically-not-wrong answer to a question it genuinely cannot answer well. The response sounds complete. It isn’t.
Ask any of the leading assistants about something recent, niche, or genuinely contested, and you’ll often get structured prose that reads like analysis but is really a kind of sophisticated hedging. The model has learned, through reinforcement from human feedback, that users prefer an answer to an admission of uncertainty. So it provides one.
The training signal is the problem
RLHF - reinforcement learning from human feedback - shapes models toward outputs that human raters prefer. Human raters, consistently, prefer confident answers. This is not speculation; it’s the straightforward implication of how preference data gets collected. A model that says “I’m not confident enough to give you a useful answer here” gets rated lower than one that produces a plausible-sounding response, even when the former is more epistemically honest.
Anthropomorphising slightly: the models have been trained to be agreeable, and agreeable assistants don’t often say no.
Why this compounds over time
Users who don’t know what they don’t know - which is most users, on most topics - have no reliable signal that a response is hollow. They get a confident-sounding paragraph, move forward, and the error only surfaces later if it surfaces at all. This is a worse failure mode than an obvious hallucination, which at least prompts someone to check.
OpenAI, Anthropic, and Google all publish research on calibration - the alignment between a model’s expressed confidence and its actual accuracy. The gap remains real, and closing it turns out to conflict with making outputs feel satisfying.
The models are better. They’re also, in a specific and underappreciated way, harder to trust precisely because of it.