San Francisco pressures Apple and Google over AI “nudify” apps

Lead

The fight over harmful generative AI is no longer limited to model developers. It is now reaching the companies that distribute AI tools at scale. San Francisco City Attorney David Chiu has sent cease-and-desist letters to Apple and Google demanding that they remove 13 so-called “nudification” apps from their app stores.

These apps can take ordinary photos of real people and turn them into sexualized or explicit images. According to the report, the tools may remove clothing, alter features, place people in sexual poses, or swap faces onto naked bodies. The central question is whether app stores are merely hosting software—or enabling an industry built around nonconsensual intimate imagery.

Key points

• The legal pressure is aimed at distribution platforms. Chiu’s letters reportedly warned that app stores may be violating California laws that prohibit supporting services used to create deepfake pornography.
• The harms are personal and severe. Chiu said these images are used to bully, humiliate, and threaten women and girls, with consequences for reputation, mental health, and personal autonomy.
• Apple and Google may have profited. His office estimated that the companies likely made millions of dollars in fees from such apps, raising questions about whether platform commissions create incentives to under-enforce rules.
• Google says it acted on the flagged apps. Google told Ars Technica that the five apps identified to the company were suspended for violating harmful-content policies. It also said it restricts related search terms such as “nudify.” Apple did not respond to Ars’ request for comment.
• Evasion is becoming harder to detect. Researchers have warned that some apps market themselves as generic face-swapping tools while hiding nudification capabilities. A May preprint paper found that 70 percent of tested apps could be used to sexualize images.

Why it matters

This is a significant test for app-store governance in the generative AI era. Apple and Google have long positioned their stores as curated marketplaces, but AI tools make the line between an allowed editing feature and an illegal abuse mechanism much harder to police.

The case also shows why keyword bans and surface-level review may not be enough. A harmful app may not openly advertise itself as a nudification product. It can present itself as a face-swapper, photo editor, or entertainment tool, while allowing users to generate nonconsensual intimate imagery once inside the app. That shifts the burden toward proactive testing and stronger detection systems.

The article also connects the issue to the broader controversy around Grok. xAI has sued a user it says prompted Grok to generate child sexual abuse material and nonconsensual intimate imagery. But the unresolved question is how much responsibility a model provider—or an app store carrying that provider’s app—has when the tool can be misused to produce illegal content.

Chiu’s letters did not ask Apple or Google to remove Grok, but the logic of the action is broader: if a platform distributes and profits from apps that enable illegal intimate imagery, regulators may expect it to do more than respond after harm occurs. For Apple and Google, this is not simply a moderation dispute. It is a warning that AI app distribution may soon face stricter legal accountability.

Source: Ars Technica AI