AI data privacy and consent

AI features tend to want more data than ordinary software: your messages, documents, photos, voice and behavior, sometimes sent to a server and sometimes used to improve the model. Privacy and consent design is about making those flows visible and giving people a real choice. That means asking at the right moment, explaining in plain words, collecting only what the feature needs, and making it as easy to say no later as it was to say yes.

Why it matters

People share more readily when they understand what happens to their data. Apple's guidelines say "People are more likely to be comfortable sharing data when they understand how it's used", and ask developers to state "whether your model uses personal information for training and improvement" (Apple HIG).

There is regulatory risk too. In 2024 the US Federal Trade Commission warned that it may be unfair or deceptive for a company to start using consumer data for AI training "and to only inform consumers of this change through a surreptitious, retroactive amendment to its terms of service or privacy policy" (FTC). Where consent is the legal basis under the EU's GDPR, Article 7(3) says "It shall be as easy to withdraw as to give consent" (GDPR Art. 7).

How to apply it

• Do ask in context. Request access to someone's inbox when they turn on an email-drafting feature, not in a wall of permissions at sign-up.
• Do say what leaves the device. Apple recommends making sure people know information may be sent to a server, "showing them what's shared", and explaining what may be stored or used for training.
• Do minimize. Process locally where possible and send only what the task needs.
• Do separate "use my data to answer me" from "use my data to train the model". They are different choices.
• Do let people see and change their settings later. PAIR advises: "Let the user know where they can see their data and where they can change data-collection settings" (PAIR Feedback + Control). Provide reset and delete options, see personalization controls.
• Don't pre-tick training opt-ins or hide the opt-out several screens deep. These are deceptive patterns.
• Don't assume a privacy policy is enough. PAIR warns: "Don't assume basic data policies are enough to protect personal privacy" (PAIR Data Collection).

Common mistakes

• Expanding data use, such as adding model training, without fresh, clear notice to existing users.
• Forgetting that outputs can leak data. Apple notes model outputs "can inadvertently contain sensitive information", for example a meeting summariser quoting a private aside to all attendees.
• Treating consent for one feature as consent for every future AI feature.
• Vague benefit statements. Apple recommends explaining benefits in a way that is "concise, specific, and easy to understand."