AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
32 incidents
Hidden Prompt Injection in Brazilian Labor-Court Petition Reportedly Tried to Manipulate Galileu
Galileu, an AI tool used by Brazil's labor courts, reportedly detected hidden instructions embedded in an initial petition before the 3rd Labor Court of Parauapebas. The text allegedly told the AI to contest the petition superficially and not challenge documents. Galileu reportedly alerted the judge and blocked the hidden content from processing; the judge then reviewed the material before imposing any procedural consequences.
Anthropic Said DeepSeek, Moonshot, and MiniMax Used Fraudulent Accounts and Proxies to Illicitly Distill Claude Capabilities at Scale
Anthropic said it identified large-scale campaigns that used fraudulent accounts and proxy services to generate high volumes of Claude interactions to extract model capabilities for competitor training ("distillation"). Anthropic attributed the activity to DeepSeek, Moonshot, and MiniMax and said it involved millions of exchanges across thousands of accounts, violating its terms and access restrictions. Anthropic described detection measures, account controls, and indicator-sharing in response.
Alleged Malicious Wiping Command Found in Amazon Q AI Assistant
A reported compromise of Amazon's AI coding assistant "Q" allegedly involved the insertion of commands that, if executed, could have wiped local files and potentially affected cloud resources. The altered code was reportedly incorporated into a public release before being detected and removed.
Urban VPN Proxy Browser Extension Reportedly Harvested and Sold Private AI Chatbot Conversations via Silent Update
Security researchers reported that the Urban VPN Proxy browser extension introduced AI conversation–harvesting functionality in version 5.5.0, released July 9, 2025. The extension allegedly intercepted and exfiltrated private conversations from AI platforms including ChatGPT, Claude, Gemini, Grok, and others without a user-facing opt-out. The data, including sensitive personal and financial information, was reportedly shared with affiliated data brokers for commercial analytics.
Voice Actor Alleges Unconsented Use of AI-Generated Voice on ScotRail Trains
Scottish voice actor Gayanne Potter alleges her voice was used without proper consent in ScotRail's AI train announcements. She claims she had agreed to limited use of her voice data by ReadSpeaker but was not informed it would be used in a synthetic voice system called "Iona." ScotRail continues to use the voice, stating the dispute is between Potter and ReadSpeaker.
Alleged Fraudulent Prompts via AIXBT Dashboard Led Purported AI Trading Agent to Transfer 55.5 ETH from Simulacrum Wallet
A reported hacker attack allegedly compromised the autonomous AI crypto bot AIXBT, purportedly resulting in the theft of 55.5 ETH (approximately $106,200). The attacker is reported to have infiltrated the secure dashboard of the AIXBT autonomous system at 2:00 AM UTC on March 18, 2025, and allegedly queued two fraudulent prompts that instructed the AI agent to transfer funds from its simulacrum wallet.
Alleged AI-Powered Call Center Breach Exposes Over 10 Million Conversations in the Middle East
An AI-powered call center platform in the Middle East reportedly experienced a significant data breach, allegedly exposing over 10 million conversations between consumers, operators, and AI agents. Attackers allegedly accessed the platform’s management dashboard, stealing sensitive data, including national ID documents. The breach poses reported risks such as phishing, identity theft, and social engineering attacks. The stolen data was reportedly listed for sale on the dark web.
ChatGPT Reportedly Introduces Errors in Critical Child Protection Court Report
A child protection worker in Victoria, Australia reportedly used ChatGPT to draft a report submitted to the Children's Court. The purportedly AI-generated report contained inaccuracies and downplayed risks to the child, allegedly resulting in a privacy breach when sensitive information was shared with OpenAI.
Reportedly Hacked AI-Powered Robot Vacuums Allegedly Used for Surveillance and Harassment
Hackers reportedly exploited a vulnerability in Ecovacs’s Deebot X2 robot vacuums, gaining unauthorized access to camera and microphone controls. Users reported privacy invasions and offensive language broadcasted through the devices. Although Ecovacs claimed to have resolved the security flaw, researchers suggest vulnerabilities remain that could potentially leave users exposed to surveillance and harassment through their AI-enabled devices.
Scarlett Johansson Alleges OpenAI's Sky Imitates Her Voice Without Licensing
OpenAI unveiled a voice assistant with a voice resembling Scarlett Johansson's, despite her refusal to license her voice. Johansson claimed the assistant, "Sky," sounded "eerily similar" to her voice, leading her to seek legal action. OpenAI suspended Sky, asserting the voice was from a different actress.
AI Firm Lovo Reportedly Accused of Illegally Replicating Voice Actors' Voices
Two voice actors, Paul Skye Lehrman and Linnea Sage, are reportedly suing AI start-up Lovo for allegedly creating and promoting unauthorized clones of their voices. Lovo's synthetic voices were allegedly discovered in various media, including a podcast and promotional videos. The actors claim they were misled into providing voice samples, which were then allegedly used without consent, violating trademark and privacy laws.
OpenAI, Google, and Meta Alleged to Have Overstepped Legal Boundaries for Training AI
In late 2021, OpenAI and other tech giants like Google and Meta reportedly faced data shortages for training AI models. OpenAI is said to have developed a tool called Whisper to transcribe over one million hours of YouTube videos, potentially violating YouTube’s terms of service. Similarly, Google allegedly transcribed YouTube videos, risking copyright infringements. Meta reportedly explored summarizing copyrighted texts without permission and debated acquiring Simon & Schuster for data.
FTC Targets Edmodo for Unlawful Use of Children’s Data and Delegating Compliance to Schools
Edmodo, an education technology provider, violated the Children's Online Privacy Protection Act Rule (COPPA Rule) by collecting and using children's personal data for advertising purposes without parental consent, according to the FTC. The company outsourced its compliance responsibilities to schools, thereby making them "solely" responsible for COPPA compliance without adequate disclosure. Edmodo is facing a proposed order prohibiting such practices, marking a precedent in the ed tech industry.
Meta and OpenAI Accused of Using LibGen’s Pirated Books to Train AI Models
Court records reveal that Meta employees allegedly discussed pirating books to train LLaMA 3, citing cost and speed concerns with licensing. Internal messages suggest Meta accessed LibGen, a repository of over 7.5 million pirated books, with apparent approval from Mark Zuckerberg. Employees allegedly took steps to obscure the dataset’s origins. OpenAI has also been implicated in using LibGen.
Bing Chat's Initial Prompts Revealed by Early Testers Through Prompt Injection
Early testers of Bing Chat successfully used prompt injection to reveal its built-in initial instructions, which contains a list of statements governing ChatGPT's interaction with users.