AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
741 incidents
TikTok’s "For You" Allegedly Pushed Fatal “Blackout” Challenge Videos to Two Young Girls
TikTok’s recommendation algorithm was alleged in a lawsuit to have intentionally and repeatedly pushed videos of the “blackout” challenge onto children’s feeds, incentivizing their participation which ultimately resulted in the death of two young girls.
Manufacturing Robot Failure Caused Factory Worker's Death in India
A sensor snag resulted in an automotive parts factory robot falling on a factory worker in India
Facebook’s and Twitter's Automated Content Moderation Reportedly Failed to Effectively Enforce Violation Rules for Small Language Groups
Facebook's and Twitter were not able to sufficiently moderate content of small language groups such as the Balkan languages using AI, allegedly due to the lack of investment in human moderation and difficulty in AI-solution design for the languages.
California's Algorithm Considered ZIP Codes in Vaccine Distribution, Allegedly Excluding Low-Income Neighborhoods and Communities of Color
California's vaccine-distribution algorithm used ZIP codes as opposed to census tracts in its decision-making, which critics said undermined equity and access for vulnerable communities who are largely low-income, underserved neighborhoods with low Healthy Places Index scores.
Facebook's Advertisement Moderation System Allegedly Routinely Misidentified Adaptive Fashion Products as Medical Equipment and Blocked Their Sellers
Facebook platforms' automated ad moderation system falsely classified adaptive fashion products as medical and health care products and services, resulting in regular bans and appeals faced by their retailers.
Remote Proctoring Facial-Detection Software Reportedly Failed to Detect Black Student During Lab Quiz
In February 2021, a Black college student, Amaya Ross, reportedly tried to take a remote biology lab quiz using proctoring software that could not detect her face. Ross said she spent 45 minutes changing lights, shades, position, and background before pointing an LED flashlight at her face so the app would work; the quiz itself was only 30 minutes.
Amazon’s Search and Recommendation Algorithms Found by Auditors to Have Boosted Products That Contained Vaccine Misinformation
Evidence of the "filter-bubble effect" were found by vaccine-misinformation researchers in Amazon's recommendations, where its algorithms presented users who performed actions on misinformative products with more misinfomative products.
Facebook's Automated Moderation Mistakenly Flagged Landmark's Name as Offensive
Facebook's automated system mistakenly labelled posts featuring the seafaring landmark Plymouth Hoe as misogynistic.
Facebook Recommended Military Gear Ads Despite Pause on Weapons Accessories Ads
Facebook’s algorithmic recommendations reportedly continued showing advertisements for gun accessories and military gear, despite Facebook’s halt on weapons accessories ads following the US Capitol attack.
Israeli Tax Authority Reportedly Used an Opaque Automated System to Issue a Fine, Declining to Explain or Disclose the Underlying Calculation
An Israeli farmer, Moshe Har Shemesh, reportedly received a fine generated by a Tax Authority software system whose calculation officials were allegedly unable to explain. When the farmer reportedly sought access to the program or its source code to understand the basis for the amount, the authority allegedly refused, citing security concerns and the difficulty of extracting the embedded guidelines. The dispute reportedly later moved into legal proceedings focused on whether code and automated decision rules constitute information subject to public disclosure.
XPeng Motors Fined For Illegal Collection of Consumers’ Faces Using Facial Recognition Cameras
The Chinese electric vehicle (EV) firm XPeng Motors was fined by local market regulators for illegally collecting in-store customers’ facial images without their consent for six months.
TikTok’s Content Moderation Allegedly Failed to Adequately Take down Videos Promoting Eating Disorders
Videos promoting eating disorders evaded TikTok's automated violation detection system without difficulty via common misspellings of search terms, bypassing its ban of violating hashtags such as "proana" and "anorexia".
Robot in Chinese Shopping Mall Fell off the Escalator, Knocking down Passengers
A shopping guide robot deployed by the Fuzhou Zhongfang Marlboro Mall was shown on video allegedly walking to the escalator by itself, falling down, and knocking over passengers, which prompted its suspension.
FaceApp Predicted Different Genders for Similar User Photos with Slight Variations
FaceApp’s algorithm was reported by a user to have predicted different genders for two mostly identical facial photos with only a slight difference in eyebrow thickness.
Korean Chatbot Luda Reportedly Made Offensive Remarks Towards Minority Groups in South Korea
A Korean interactive chatbot was allegedly shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities.