San Francisco City Attorney Sues Operators of AI Deepfake Pornography Websites for Violations of State and Federal Law
Description
In August 2024, the San Francisco City Attorney’s Office filed a first-of-its-kind lawsuit against 16 websites accused of using generative AI to create and distribute nonconsensual pornographic deepfakes, including images of minors. The sites, which had over 200 million visits in six months, invited users to upload real images to generate explicit fakes. By March 2025, 11 of the targeted websites had shut down following legal pressure.
Actors
Alleged Deployers
- operators-of-16-deepfake-pornography-websites
- synthetic-media-creators
- deepfake-creators
Alleged Developers
- deepfake-technology-developers
- synthetic-media-generation-technology-developers
Harmed Parties
- minors-depicted-in-deepfake-pornography
- women
- women-and-girls
- girls
- minors
- privacy
- victims-of-deepfake-abuse
- victims-of-deepfake-child-abuse
- victims-of-non-consensual-deepfakes
- people-depicted-in-non-consensual-intimate-imagery
Classification
Risk Domain
Malicious Actors & MisuseRisk Subdomain
4.3. Fraud, scams, and targeted manipulation
MIT Risk Profile
Source
Related Incidents (Malicious Actors & Misuse)
View all →Female Students at Westfield High School in New Jersey Reportedly Targeted with Deepfake Nudes
Oct 2023
ChatGPT Was Alleged to Have Aided Planning of Florida State University Mass Shooting
Apr 2025
Ongoing Purported AI-Assisted Identity Fraud Enables Unauthorized Access to Western Companies by North Korean IT Workers
Research Papers on This Topic
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Regulating ChatGPT and other Large Generative AI Models
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack