Hallucination

A hallucination is output from a generative model that reads as correct but isn't: an invented citation, a wrong date, a policy that doesn't exist. NN/g defines it as output that "seems plausible but is incorrect or nonsensical" (Laubheimer, 2025). NIST prefers the term confabulation, "the production of confidently stated but erroneous or false content", and notes it is known colloquially as hallucination (NIST AI 600-1).

You can't design hallucinations away entirely. A 2025 paper by Kalai and colleagues argues that models guess when uncertain partly because training and benchmark scoring reward confident answers over admitting uncertainty (Kalai et al.). The design job is to reduce how often they happen and limit the damage when they do.

Why it matters

Hallucinations are dangerous because they look exactly like correct answers. The same fluent tone covers both, so users have no built-in signal to tell them apart. Apple's guidelines warn that the model "may convincingly present the information as factual, even when it's not" (Apple HIG).

The cost can be real. In 2024, Air Canada was ordered to compensate a passenger after its website chatbot wrongly told him he could claim a bereavement fare after travel. The tribunal called the airline's argument that the chatbot was responsible for its own actions "a remarkable submission" (Daily Hive). If your product says it, users and courts will treat it as your claim.

How to apply it

• Do scope what you ask the model to generate. Apple advises avoiding requests for factual information "unless you're confident the model has access to verified and up-to-date information for the task."
• Do ground answers in retrieved sources and show them, so people can check claims. See citations and grounding.
• Do signal uncertainty where it matters, for example a meeting-notes summariser that flags low-confidence action items rather than stating them flatly. NN/g suggests first-person hedging and confidence indicators in high-stakes domains.
• Do make correction cheap: edit, retry and "report a problem" controls next to the output.
• Don't rely on a generic footer disclaimer. NN/g recommends contextual warnings at the moment uncertainty matters instead of small print.
• Don't use generated content where a wrong answer could cause harm without human review, such as dosage, legal or refund policy answers.

Common mistakes

• Treating hallucination as a model problem only. Interface choices (tone, missing sources, no edit path) decide whether an error gets caught.
• Showing citations that don't actually support the sentence they sit next to, which adds false credibility.
• Letting an agent act on a hallucinated fact, for example booking travel based on an invented fare rule, without a confirmation step.
• Testing only happy paths. Probe with vague, out-of-scope and adversarial prompts, where fabrication is more likely.