Why a model makes things up, and what actually reduces it

One process produces the right answer and the wrong one

At each step a language model estimates which token is likely to follow what has been written, picks one, and repeats. Its training objective is plausibility of the continuation, not correspondence with the world. Preference training then pushes it toward answers people rate as helpful and clear. Neither stage measures truth directly. The second rewards it only where a rater noticed the error, which means it rewards truth best exactly where mistakes are easiest to spot, and worst everywhere else.

There is no separate store of facts to consult. What the model knows is smeared across billions of weights, tangled with grammar and style; there is no row to look up, and no lookup that can fail. Where the data makes the true continuation the likely one — the Tigris runs through Baghdad — you get the truth. Where nothing pins the answer down, say the DOI of one paper, the same machinery runs on and produces something shaped like a DOI. Nothing separates the two cases.

This is why “hallucination” is a poor name: it suggests a lapse in an otherwise reliable process, when it is closer to normal operation on inputs where fluency and accuracy diverge. Recent work on evaluation makes the further argument that the behaviour is trained in, since a benchmark awarding zero for “I don't know” and full marks for a lucky guess makes guessing the score-maximising strategy.

Where it goes wrong, predictably

References are the worst case, because their form is so regular. Author, year, title, journal, volume, pages, identifier: a model can produce a flawlessly formatted citation to a paper that was never written, and the formatting is what makes it convincing. In Mata v. Avianca (2023) a New York lawyer filed a brief citing cases that did not exist, and was sanctioned for it. Nor is that an outlier. A 2024 Stanford RegLab study of more than 800,000 verifiable questions about US federal cases measured hallucination rates of 58% for GPT-4 and 88% for Llama 2.

Numbers and dates fail for a related reason: a digit string is cheap to produce and carries no internal check, and the same holds for transliterated names, where several near-identical spellings compete.

Then there is anything niche. The thinner the data on a subject, the more the model interpolates between things it half-knows. For an Arabic-speaking reader this bites harder than the English-language discussion suggests: local institutions, Iraqi-dialect material and Arabic sources are thinly represented, for reasons we have written about separately, so a model that can paraphrase a widely reproduced US federal statute will invent an Iraqi regulation outright — and the Arabic answer will read no less fluently for it.

Finally, anything your phrasing presupposes. Ask why X causes Y and the likely continuation is an explanation. A false premise buried in a question is one of the most reliable ways to extract a fabrication: you handed the model a frame and it completed it.

Confidence is a writing style, not a signal

The assured tone comes from the register of the text the model learned to imitate, and from preference tuning that rewarded direct, useful-sounding prose. Fluency, specificity and an air of authority are produced upstream of any question about whether the answer is right. Developers have themselves reported that preference training can leave a model's probability estimates less calibrated than the base model's.

This is also why “are you sure?” is close to useless: it gives the model one more turn in a context that now carries a mild signal of disapproval, and preference-trained models tend to move toward the user. Studies of intrinsic self-correction — revising with no new information available — find answers often get worse rather than better.

Do not overstate this. Ongoing research suggests internal states carry some signal about whether an answer is correct, and token-level probabilities are not pure noise; but that signal is not what reaches you as tone. One check costs nothing and needs no source at hand: ask the same question in a fresh session two or three times. Divergent answers are strong evidence of guessing; matching answers are weak evidence of anything.

What actually reduces it, strongest first

Give it the text. The largest improvement available to you is moving the task from recall to reading: paste the contract, the paper, the log, the standard. Retrieval systems do this at scale, and they work not because of clever prompting but because the answer is now in the context window. It is not free: when retrieval returns passages that do not contain the answer, models still tend to answer, which is why “if the source does not say, say so” earns its place here and nowhere else. Sourced answering has failure modes of its own, covered elsewhere.

Ask for citations you can open, then open them; the value is entirely in the opening. Stronger still is quote-then-answer: require the exact sentence from the source before the conclusion drawn from it. A fabricated quote against a document you hold is visible at once, in a way a fabricated conclusion is not.

Split the question. One question hiding five steps returns a confident paragraph with the wrong step buried inside it; broken into steps whose intermediate values you can see, the same work becomes checkable.

Let it think longer when the answer has to be right. Models that reason at length before answering do measurably better where the answer can be derived: arithmetic, multi-step deduction, internal consistency, noticing that a premise is false. Extra reasoning cannot manufacture knowledge that was never in the training data, and vendors' own reports show these models still fabricate.

What does not help, however often it is repeated

Telling a model not to hallucinate buys much less than it promises. What it reliably changes is register: the output becomes more cautious, more qualified, more sprinkled with “approximately”, which yields hedged fabrications rather than fewer of them. The instructions that earn their place name a checkable condition for abstaining, such as the absence of the claim from a source in front of the model, rather than asking it to judge its own knowledge.

Setting temperature to zero removes variety, not error. You get the most likely continuation every time; if it is wrong, you get the same wrong answer nearly every time (hosted inference is not bit-for-bit deterministic even at zero), which is easily mistaken for reliability. Self-reported confidence percentages are systematically overconfident and cluster at a handful of round values whatever the question, so they separate a solid answer from a guess far more weakly than the number's precision implies. Read one as a mood, never as a probability.

Threats, offers of payment and elaborate politeness reliably change the style of an answer. Measured effects on accuracy are small, inconsistent between models and tasks, and have not held up well on re-testing: nothing you would build a workflow on.

Reduced, not solved

No published method removes the behaviour, and there are formal arguments that none can: a system required to answer a question its training data does not determine will be wrong some fraction of the time, and the only levers are how large that fraction is and whether you find out. Use these systems where verification is cheap: code you can run, documents in front of you. Be far more careful where being wrong is expensive: medicine, law, finance, anything you sign your name to.

Firas Brain, our own tool for this, answers from documents you supply and cites them, which shortens the checking and does not remove it.