AI mental models

An AI mental model is a user's working theory of an AI product: what it knows, how it decides, when it is likely to be wrong, and whether it learns from them. Google's PAIR guidebook defines a mental model in general as "a person's understanding of how something works and how their actions affect it." AI products are hard to model because they are probabilistic, can change over time, and often talk like people.

Why it matters

Mental models decide how much people rely on a system and when. PAIR notes that "Mismatched mental models can lead to unmet expectations, frustration, misuse, and product abandonment," and that "When users can't accurately map the system's abilities, they may over-trust the system at the wrong times."

Common mismatches in AI products include:

• The search engine model. Users may assume a chat assistant looks facts up, when it may be generating text without retrieval, which makes hallucinations more convincing.
• The person model. Conversational interfaces suggest human understanding. PAIR warns that people "are more likely to have unachievable expectations for products that they assume have human-like capabilities."
• The deterministic model. People expect the same input to give the same output. Apple's generative AI guidelines point out that with generative AI, small changes to inputs, or even the same input given multiple times, often produce very different outcomes.
• The always-learning model. Users assume every correction trains the system immediately, or never does, when the truth is usually somewhere in between.

How to apply it

• Do start from existing mental models. PAIR suggests looking at how people solve the problem today, since that will shape their first assumptions.
• Do state limits plainly and show how to get good results. Apple says clarifying capabilities and limitations "helps people establish a mental model of your feature."
• Do use factual, non-human language for explanations. NN/g recommends "This answer is based on the following source: [link]" over phrasing like "I thought about your problem and searched the internet."
• Do show that feedback has an effect. Apple notes that a clear signal when people adjust output "helps people build an accurate mental model of your feature."
• Do tell people when a limitation is fixed. Apple's machine learning guidelines point out that frequent users learn to avoid failing interactions and need to know when they can try again.
• Don't promise "AI magic." PAIR says hiding how a product works can set users up for confusion and broken trust.

Common mistakes

• Giving the assistant a persona and first-person voice without considering the expectations it creates.
• Assuming one onboarding screen fixes the model. Mental models keep changing as people use the product, so the product's cues need to stay consistent over time.
• Explaining the underlying technology when users need to know what to expect from it.
• Ignoring research. Ask users to describe how they think the system works; their answers reveal the mismatches to design for.