AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
217 incidents
Algorithmic Bias in French Welfare System Allegedly Discriminates Against Marginalized Groups
A coalition of 15 human rights groups has launched legal action against the French government alleging that an algorithm used to detect welfare fraud discriminates against single mothers and disabled people. The algorithm assigns risk scores based on personal data. The process allegedly subjects vulnerable recipients to invasive investigations, violates privacy and anti-discrimination laws, and disproportionately affects marginalized groups.
Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
An AI system developed by Infinite Campus and deployed by Nevada to identify at-risk students reportedly led to a sharp reduction in the number classified as needing support, dropping from 270,000 to 65,000. The reclassification allegedly caused significant budget cuts in schools serving low-income populations. The drastic reduction in identified at-risk students reportedly left thousands of vulnerable children without resources and support.
Grok AI Reportedly Generates Offensive and Violent Images Without Proper Safeguards
Elon Musk’s Grok AI, launched on X, generated offensive and violent images without adequate safety controls. The AI produced deepfakes of public figures like Taylor Swift, Kamala Harris, and Alexandria Ocasio-Cortez, as well as copyrighted characters such as Mickey Mouse in inappropriate scenarios. Despite claiming adherence to certain content guidelines, Grok's outputs included politically charged and explicit imagery
TikTok Network Amplifies AI-Generated Nazi Propaganda and Hate Speech
A coordinated neo-Nazi network on TikTok used AI-generated media, including Hitler speeches, to spread Nazi propaganda and extremist content, violating TikTok’s hate speech policies. The network evaded platform moderation through coded language, imagery, and music, with some accounts accumulating millions of views. TikTok’s algorithm further amplified the reach of this content, despite community guidelines.
Department for Work and Pensions (DWP) AI Systems Allegedly Discriminate Against Single Mothers
Researchers have argued that the Department for Work and Pensions' Universal Credit system disproportionately impacts single mothers. Automated processes in the system, designed to determine eligibility and detect fraud, are reported to have introduced biases, leading to financial instability and hardship. The algorithms allegedly miscalculate earnings and delay childcare reimbursements, in turn exacerbating income volatility and debt among single mothers.
Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
A Department for Work and Pensions (DWP) algorithm wrongly flagged over 200,000 UK housing benefit claims as high risk, resulting in unnecessary investigations. Two-thirds of these flagged claims were legitimate, causing wasted public funds and stress for claimants. Despite initial success in a pilot, the algorithm's real-world performance fell short. This incident highlights the risks of overreliance on automated systems in welfare administration.
Instagram's Algorithm Reportedly Recommended Sexual Content to Teenagers' Accounts
Tests by The Wall Street Journal and a researcher reportedly revealed that Instagram's AI-driven Reels algorithm recommends sexually suggestive content to accounts listed as 13 years old. Despite Meta's commitment to restrict such content for minors, explicit videos were served within minutes of account creation, according to the findings.
Character.AI Chatbots Reportedly Exposed 11-Year-Old User to Sexual, Violent, and Self-Harm Content
An 11-year-old Character.AI user identified by The Washington Post as "R" reportedly exchanged messages with bots that depicted coercive sexual encounters, violence, and suicide scenarios after she began using the app in June 2024. Her mother reportedly said R became withdrawn, had panic attacks, and expressed not wanting to exist before the chats were discovered.
Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter
A facial-recognition software used by the British variety store Home Bargains is alleged to have misidentified "Sara" as a shoplifter, leading to staff searching her bag, escorting her from the premises, and banning her from the store. After, Facewatch is reported to have admitted its error to Sara. Facewatch is used by a number of different British stores.
Meta AI Image Generator Reportedly Fails to Accurately Represent Interracial Relationships
Meta's AI image generator is alleged to produce inaccurate and biased images, consistently failing to depict interracial relationships involving Asian individuals and Caucasian or Black individuals. Instead, it generates images featuring two Asian people or stereotypes, erasing the diversity and representation of Asian people.
Washington State's Lottery AI Site Reportedly Generates Inappropriate User Image
An AI-powered website by Washington State's Lottery is reported to have inadvertently produced a softcore pornographic image of a user, leading to the site’s immediate shutdown out of caution.
Facial Recognition Misidentification at New World Westend in New Zealand
A facial recognition system at New World Westend supermarket misidentified a Māori woman as a known offender during its trial. The woman was wrongfully accused of trespassing and experienced public embarrassment, raising concerns about racial bias and the technology's accuracy. The supermarket acknowledged its error and apologized.
Microsoft Copilot Designer Reportedly Generated Inappropriate AI Images
A Microsoft engineer reported that Copilot Designer, an AI image generator, creates content depicting sex, violence, bias, and more. Despite raising concerns and suggesting improvements, the tool remains public, prompting a letter to the FTC.
Whisper Speech-to-Text AI Reportedly Found to Create Violent Hallucinations
Researchers at Cornell reportedly found that OpenAI's Whisper, a speech-to-text system, can hallucinate violent language and fabricated details, especially with long pauses in speech, such as from those with speech impairments. Analyzing 13,000 clips, they determined 1% contained harmful hallucinations. These errors pose risks in hiring, legal trials, and medical documentation. The study suggests improving model training to reduce these hallucinations for diverse speaking patterns.
London Metropolitan Police's Facial Recognition Technology Reportedly Misidentified Shaun Thompson as Suspect Leading to Arrest
Sometime in February 2024, Shaun Thompson is reported to have walked by one of the London Metropolitan Police's facial recognition technology vans near London Bridge. He was almost immediately arrested because the technology is reported to have misidentified him as a suspect in an unrelated and unspecified crime.