Automation bias

Automation bias is what happens when people treat an automated recommendation as a shortcut for their own judgment. It was studied for decades in cockpits, control rooms and clinical decision support, long before chat assistants. The same pattern now shows up whenever someone accepts an AI summary, code suggestion or extracted figure without checking it.

Researchers describe two kinds of error. An omission error is missing a problem because the system didn't flag it. A commission error is following the system's advice when it is wrong. Parasuraman and Manzey's review found that automation bias "results in making both omission and commission errors when decision aids are imperfect" (Parasuraman & Manzey, 2010).

Why it matters

The same review found that automation bias "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams." In other words, you can't fix it with an onboarding tooltip. It has to be handled in the design of the workflow.

It is also the main way hallucinations cause harm. A wrong answer that nobody acts on is harmless. A wrong answer pasted into a report, a contract or a patient note because it looked authoritative is not. Lee and See link this to trust calibration: overtrust, where trust exceeds the system's capabilities, may lead to misuse (Lee & See, 2004).

How to apply it

• Do make verification part of the flow, not an optional extra. A meeting-notes summariser can link each action item to the transcript moment it came from.
• Do highlight what the system is unsure about, and what it may have missed, so omission errors are less likely.
• Do ask for confirmation before significant or irreversible actions. Apple's guidelines recommend asking "for confirmation before performing a significant action on someone's behalf" (Apple HIG).
• Do consider letting users form their own view first in high-stakes review tasks, then reveal the AI suggestion.
• Don't pre-fill decisions with AI output in a way that makes accepting the default the path of least effort for consequential choices.
• Don't assume explanations will help. One CHI study found explanations increased acceptance of AI recommendations "regardless of its correctness" (Bansal et al., 2021).

Common mistakes

• Treating "a human reviews it" as a safeguard without checking whether reviewers actually catch errors. A human-in-the-loop who approves everything is not a control.
• Measuring success by acceptance rate alone. High acceptance can mean the AI is good or that people stopped checking.
• Overloading reviewers. Parasuraman and Manzey link complacency to multiple-task load, so a reviewer juggling other work is more likely to miss AI errors.
• Using polished, confident formatting for low-confidence output, which invites uncritical acceptance.