Calibrated trust

Calibrated trust is the goal of trust design in AI products: people should rely on the system when it is likely to be right and check or override it when it is likely to be wrong. The idea comes from human factors research on automation. Lee and See define calibration as "the correspondence between a person's trust in the automation and the automation's capabilities" (Lee & See, 2004).

The target is not maximum trust. Google's PAIR guidebook puts it plainly: "Users shouldn't implicitly trust your AI system in all circumstances, but rather calibrate their trust correctly" (PAIR).

Why it matters

Miscalibration fails in two directions. Lee and See describe overtrust as "poor calibration in which trust exceeds system capabilities", and distrust as trust that "falls short of automation capabilities". Overtrust may lead to misuse and distrust may lead to disuse. In an AI product, overtrust looks like copying a drafted contract clause without reading it, or accepting a hallucinated figure. Undertrust looks like users re-doing work the assistant got right, or abandoning a feature that would have saved them time.

Good calibration also has to be specific. A coding assistant might be very reliable at renaming variables and much less reliable at security-sensitive logic. Users need to learn where it is strong and where it is weak, not one overall verdict.

How to apply it

• Do tell people what the system can do and how well. Microsoft's HAX guidelines open with "Make clear what the system can do" (G1) and "Make clear how well the system can do what it can do" (G2).
• Do show uncertainty where it changes decisions, such as confidence indicators on extracted invoice fields, and only where users can act on it. PAIR advises showing confidence when it improves decision making.
• Do make verification cheap: sources, previews and diffs before an agent applies changes.
• Do let trust be earned over time, for example by starting an agent in suggest-only mode before allowing it to act.
• Don't assume explanations calibrate trust. In a CHI 2021 study, explanations "increased the chance that humans will accept the AI's recommendation, regardless of its correctness" (Bansal et al.).
• Don't use confident, human-like language for outputs the system is unsure about.

Common mistakes

• Measuring trust with satisfaction scores alone. High trust is a problem if the system is often wrong. Compare reliance with actual accuracy.
• Marketing copy that oversells the capability the interface then has to walk back.
• One-size disclaimers that users learn to ignore, instead of signals tied to specific outputs.
• Ignoring undertrust. If people never use a reliable feature, that is a calibration failure too.