Why San Francisco is demanding a purge of AI apps

The city of San Francisco has flagged these 13 apps for enabling what is often called nudification or deepfake pornography. These apps take a real photograph of a person—usually a woman or a child—and use AI to manipulate the image into a sexualized or nude depiction without that person’s permission. While both Apple and Google have policies that theoretically ban such content, these apps often bypass those filters by marketing themselves as harmless face-swapping or creative photo-editing tools. Because these apps charge subscription fees or offer paid upgrades, the tech giants take a financial cut of every transaction. The City Attorney is arguing that by hosting these apps and collecting profits from them, Apple and Google are essentially facilitating illegal behavior.

At the center of this problem is the way modern AI is built. Most of these apps use generative AI, which is a type of software trained on massive datasets to recognize patterns in images and predict how to create new ones from scratch. In the case of face-swapping or image-generation tools, the AI learns what human bodies, clothing, and textures look like. It can then take a reference photo and, with just a few clicks from a user, blend that face or body into a completely different, artificial context. Because the core engine of the app is technically just an image manipulator, it is easy for developers to hide its harmful capabilities. A developer might label the app as a tool to put your face on a superhero or to change your hair color, but the underlying software remains capable of much more. This makes it difficult for human moderators at Apple and Google to identify the app as harmful just by looking at the interface. They are essentially blind to the true purpose of the software until after users begin reporting illegal activity or researchers uncover the misuse.

The city’s ultimatum strikes at the heart of how we think about responsibility online. For years, tech platforms have marketed their app stores as curated, safe spaces where users can trust the software they download. However, as AI tools become easier to build and deploy, it is becoming clear that current screening methods are failing. This creates a cycle where victims, including minors, are targeted by harassment that can be weaponized in seconds. If the city succeeds in forcing these changes, it could set a new precedent: that the giant companies providing the digital storefronts are legally accountable for the software they host, regardless of how cleverly that software is marketed. It raises a difficult question for all of us: if the technology is designed to be flexible, can it ever truly be kept safe by a checklist of rules?