Apple and Google have been ordered by San Francisco officials to remove AI-powered “nudify” apps from their app stores, as regulators intensify pressure on major platforms over tools that can create nonconsensual intimate deepfake images.
San Francisco Targets App Store Gatekeepers
The city’s action focuses not only on the app developers, but also on Apple and Google as platform operators.
TechCrunch reported that San Francisco said Apple and Google continued to host and make money from apps despite California laws targeting nonconsensual deepfake pornography. San Francisco City Attorney, David Chiu, shared that Apple and Google are profiting from apps that exploit women and girls by generating nonconsensual intimate deepfakes. Chiu added that companies have a responsibility to ensure apps on their platforms do not facilitate sexual abuse.
Letters Warn of Legal Exposure
The legal notices argue that the companies have been warned before.
WIRED reported that the letters said the companies should stop profiting from harmful technology and that California laws prohibit supporting services that create deepfake pornography.
The companies could face financial penalties if they fail to act. Chiu’s letters warned Apple and Google they could face civil penalties and asked them to contact the city within 28 days. Chiu’s office may consider legal options if Apple and Google do not remove the apps and strengthen their screening systems.
Researchers Found Large App Store Exposure
Independent researchers have repeatedly flagged the problem. The Tech Transparency Project warned Apple and Google in January and April about “dozens of apps” that sold deepfake nonconsensual intimate images through payments processed by the companies.
The Tech Transparency Project uncovered around 100 apps across the App Store and Play Store, with estimated combined downloads of about 480 million and possible combined revenue of around $120 million.
The problem also extends beyond apps openly marketed as nudification tools. Cornell University and Georgetown University researchers identified 420 face-swapping apps across Apple and Google app stores and found that 70% of the 155 tested apps could be used to create face swaps with nude images. The researchers described these face-swap apps as “dual-use” tools that may appear benign while still being capable of creating harmful content.
Apple and Google Say They Are Acting
Both companies said they have taken enforcement action. Apple said nudify apps are forbidden from the App Store and that it removed three apps while moving to terminate related developer accounts. Apple spokesperson Adam Dema said that the App Store strictly prohibits apps designed to generate, distribute or consume pornography.
Google also pointed to prior removals. Google said all five Play apps referenced in Chiu’s letter had been suspended and that it had suspended hundreds of violating apps. Google spokesperson Dan Jackson said Google Play does not allow apps containing sexual content and has restricted related search terms such as “nudify”.
Deepfake Abuse Becomes an App Store Test
The San Francisco demand raises a broader question for AI governance: whether app stores can treat nonconsensual deepfake tools as ordinary policy violations, or whether they now face legal responsibility when such tools pass through their marketplaces and payment systems.
The issue is especially urgent because harmful image-generation tools no longer require technical skill. A public photo can be enough to target a person, and the resulting images can damage reputation, autonomy and mental health. For Apple and Google, the case is a warning that app store safety claims will be judged not only by written rules, but y how quickly platforms remove tools that enable abuse.
