San Francisco Takes Action Against AI Nudity Apps

San Francisco is taking a strong stand against technology that creates fake, sexualized images. This week, the city’s attorney general, David Chiu, sent official letters to Apple and Google. These letters, known as cease-and-desist orders—which are formal demands to stop a specific activity—require both companies to remove 13 different apps from their digital stores. These apps use artificial intelligence (AI) to transform normal photos of people into explicit images without their consent.

According to a report by Wired, these tools are becoming very easy to find and use. They allow people to remove clothing from photos, change a person’s features, or place someone’s face onto a different body. The impact of this technology is severe. David Chiu explained that these images are frequently used to bully, humiliate, and threaten women and girls. In his comments to Wired, Chiu said his office was “absolutely horrified” by how common these tools have become. He noted that the technology causes deep harm to a person’s mental health and their reputation, and in some cases, it has even led victims to feel suicidal.

One of the main concerns raised by the city is that Apple and Google have been making money from these apps. The attorney general’s office estimated that the two tech giants have likely made millions of dollars in fees by allowing these programs to stay in their stores. Chiu argued that instead of taking strong action, the companies have allowed these apps to remain, which he called completely unacceptable. He believes that the companies have a responsibility to do more to stop the spread of such harmful content.

While the specific names of the 13 apps were not made public to prevent more people from searching for them, Wired reported that some of these tools were very popular. One app had already been downloaded more than a million times. It reportedly advertised features that allowed users to create sexualized images of women or make videos that were described as "free and uncensored." This type of content is a violation of California laws, which prohibit services that help create deepfake pornography—a term used for AI-generated images or videos that look real but are fake and intended to deceive or harm.

In response to the situation, Google provided a statement to Ars Technica. A spokesperson for the company, Dan Jackson, said that the five apps identified by the attorney general had already been suspended from the Google Play store. Jackson explained that Google does not allow apps with sexual content and that they are working to improve how they find and remove such programs. He added that the company has already suspended hundreds of apps that break their rules and has also restricted search terms like “nudify” to make it harder for users to find these tools.

This situation highlights a growing problem in the tech world. As AI technology becomes more powerful, it is also becoming easier for bad actors to use it for harmful purposes. Many experts agree that the responsibility lies with the platforms that host these apps. By allowing these tools to exist, app stores have become part of a public nuisance—a situation that causes harm to the community. The actions taken by San Francisco show that local governments are no longer willing to wait for tech companies to solve these problems on their own.

Looking ahead, it remains to be seen how Apple and Google will change their policies to prevent similar apps from appearing in the future. While Google has taken steps to remove the specific apps mentioned by the attorney general, the challenge of monitoring millions of apps is significant. For now, the move by San Francisco serves as a warning that the era of ignoring these harmful AI tools is coming to an end. The goal is to protect the privacy and safety of individuals, especially those who are most vulnerable to online harassment and abuse.