EN ES
Contact

← All insights

A person analyzing data on a computer screen.

Plate · Mist over forest — confidence obscuring the ground truth

Investigation

The confident answer problem: what the evidence says

Language models sound equally sure when they are right and when they are wrong. The published research on hallucination and calibration shows why confidence is the wrong signal to trust — and what to ask for instead.

November 12, 2024 · Enpath · 3 min read

The most dangerous property of a modern language model is not that it is sometimes wrong. It is that it is wrong in the same tone of voice it uses when it is right.

That claim is easy to assert. This piece walks through the published evidence for it, because the evidence is what makes it a design constraint rather than an opinion.

Fluency and truth come apart

In 2021, researchers at Oxford and OpenAI built TruthfulQA, a benchmark of questions designed so that imitating common human text produces false answers. The best model they tested was truthful on 58% of questions; humans managed 94% (Lin et al., 2021). The striking finding was not the error rate — it was that larger, more fluent models were often less truthful on these questions, because they had learned popular misconceptions more thoroughly.

The pattern has persisted as models improved. OpenAI’s own SimpleQA benchmark, published in 2024, found that frontier models answered short factual questions incorrectly a substantial share of the time — while rarely declining to answer (OpenAI, 2024). The models knew a great deal. They did not reliably know what they didn’t know.

The stakes are not hypothetical

In domains where wrong answers carry real cost, the measured rates are sobering. A Stanford study of legal questions found that leading models hallucinated — inventing cases, misstating holdings, or attributing rulings to the wrong court — on a majority of the specific legal queries tested, with rates between roughly 58% and 82% depending on the model and task (Dahl et al., 2024). Courts have since sanctioned lawyers in several countries for filing briefs citing cases that do not exist.

The same failure travels everywhere expertise is scarce. A municipal planner in a mid-sized city — in Colombia, in Indonesia, in Poland — asking a model about zoning precedent or flood-return periods has less ability to catch a confident fabrication than the lawyer did, and often no institutional reviewer behind them. The confident answer problem lands hardest exactly where the analytical safety net is thinnest.

Can a model know what it doesn’t know?

Partly — and this is the encouraging half of the evidence. Research at Anthropic found that large models can be trained to produce reasonably calibrated self-assessments: asked to estimate the probability that their own answer is correct, their estimates track reality better than chance, though imperfectly and unevenly across domains (Kadavath et al., 2022).

So the raw material for honesty exists. What is missing is the practice of surfacing it. Most deployed systems still present every answer at the same register — no confidence, no sources, no flag on the claims most likely to be invented. The uncertainty is in the machine; it just never reaches the person deciding.

What this means for anyone who decides

Three rules fall straight out of the evidence:

  1. Never use fluency as a proxy for reliability. The research shows they are uncorrelated at best.
  2. Demand calibrated confidence, claim by claim. Not a disclaimer at the bottom — a signal on each factual assertion, backed by a source that can be opened.
  3. Treat unsourced specifics as unverified. Names, numbers, citations, and quotes are precisely the details models most confidently invent.

This is why the Enpath Standard asks its seven questions of every answer. The confident answer problem cannot be solved by making models sound more careful. It is solved by changing what an answer is — from a fluent paragraph into a structure that carries its evidence, its uncertainty, and a person who stands behind it.

Sources