AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
244 incidents
Amazon's Experimental Hiring Tool Allegedly Displayed Gender Bias in Candidate Rankings
Between 2014 and 2017, Amazon reportedly developed an AI-powered recruiting tool to score job applicants, trained on a decade of resumes purportedly drawn largely from men. Media reports say the system learned to favor male candidates, penalizing terms like "women's" and graduates from certain all-women's colleges. Efforts to remove these biases reportedly did not guarantee fairness, and the project was ultimately abandoned. Amazon reportedly states recruiters never solely relied on the tool.
Common Biases of Vector Embeddings
Researchers from Boston University and Microsoft Research, New England demonstrated gender bias in the most common techniques used to embed words for natural language processing (NLP).
Northpointe Risk Models
An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.
COMPAS Algorithm Reportedly Performs Poorly in Crime Recidivism Prediction
Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a recidivism risk-assessment algorithmic tool used in the judicial system to assess likelihood of defendants' recidivism, is found to be less accurate than random untrained human evaluators.
Chinese Insurer Ping An Employed Facial Recognition to Determine Customers’ Untrustworthiness, Which Critics Alleged to Likely Make Errors and Discriminate
Customers’ untrustworthiness and unprofitability were reportedly determined by Ping An, a large insurance company in China, via facial-recognition measurements of micro-expressions and body-mass indices (BMI), which critics argue was likely to make mistakes, discriminate against certain ethnic groups, and undermine its own industry.
Google Images Showed Racially Biased Results for 'Professional' Hairstyles
Google Images reportedly showed disparate search results along racial lines, featuring almost exclusively white women for "professional hairstyles" and Black women for "unprofessional hairstyles" prompts.
Biased Google Image Results
On June 6, 2016, Google image searches of "three black teenagers" resulted in mostly mugshot images whereas Google image searchers of "three white teenagers" consisted of mostly stock images, suggesting a racial bias in Google's algorithm.
Microsoft's TayBot Allegedly Posts Racist, Sexist, and Anti-Semitic Content to Twitter
Microsoft's Tay, an artificially intelligent chatbot, was released on March 23, 2016 and removed within 24 hours due to multiple racist, sexist, and anti-semitic tweets generated by the bot.
Is Pokémon Go racist? How the app may be redlining communities of color
Through a crowdsourcing social media campaign in 2016, several journalists and researchers demonstrated that augmented reality locations in the popular smartphone game Pokemon Go were more likely to be in white neighborhoods.
Twitter Recommender System Amplified Right-Leaning Tweets
Twitter’s “Home” timeline algorithm was revealed by its internal researchers to have amplified tweets and news of rightwing politicians and organizations more than leftwing ones in six out of seven studied countries.
Uber's Surge Pricing Reportedly Offered Disproportionate Service Quality along Racial Lines
Uber's surge-pricing algorithm which adjusts prices to influence car availability inadvertently caused better service offering such as shorter wait times for majority white neighborhoods.
Predictive Policing Biases of PredPol
Predictive policing algorithms meant to aid law enforcement by predicting future crime show signs of biased output.
Images of Black People Labeled as Gorillas
Google Photos image processing software mistakenly labelled a black couple as "gorillas."
Google’s YouTube Kids App Presents Inappropriate Content
YouTube’s content filtering and recommendation algorithms exposed children to disturbing and inappropriate videos.
Gender Biases of Google Image Search
Google Image returns results that under-represent women in leadership roles, notably with the first photo of a female "CEO" being a Barbie doll after 11 rows of male CEOs.