AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
271 incidents
Chinese Insurer Ping An Employed Facial Recognition to Determine Customers’ Untrustworthiness, Which Critics Alleged to Likely Make Errors and Discriminate
Customers’ untrustworthiness and unprofitability were reportedly determined by Ping An, a large insurance company in China, via facial-recognition measurements of micro-expressions and body-mass indices (BMI), which critics argue was likely to make mistakes, discriminate against certain ethnic groups, and undermine its own industry.
Amazon Flex Drivers Allegedly Fired via Automated Employee Evaluations
Amazon Flex's contract delivery drivers were dismissed using a minimally human-interfered automated employee performance evaluation based on indicators impacted by out-of-driver's-control factors and without having a chance to defend against or appeal the decision.
Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
The Intelligence-Led Policing model rolled out by the Pasco County Sheriff’s Office was allegedly developed based on flawed science and biased data that also contained sensitive information and irrelevant attributes about students, which critics said to be discriminatory.
Tinder's Personalized Pricing Algorithm Found to Offer Higher Prices for Older Users
Tinder’s personalized pricing was found by Consumers International to consider age as a major determinant of pricing, and could be considered a direct discrimination based on age, according to anti-discrimination law experts.
UK Home Office's Sham Marriage Detection Algorithm Reportedly Flagged Certain Nationalities Disproportionately
UK Home Office's opaque algorithm to detect sham marriages flagged some nationalities for investigation more than others, raising fears surrounding discrimination based on nationality and age.
Oracle's Algorithmic Data Processing System Alleged as Unlawful and Violating Privacy Rights
Oracle's automated system involving algorithmic data processing was alleged in a lawsuit to have been unlawfully collecting personal data from millions of people and violating their privacy rights.
Uber Deployed Secret Program To Deny Local Authorities Rides
Uber developed a secret program "Greyball" which prevented known law enforcement officers in areas where its service violated regulations from receiving rides.
US DHS’s Opaque Vetting Software Allegedly Relied on Poor-Quality Data and Discriminated against Immigrants
US Citizenship and Immigration Services (USCIS)’s ATLAS software used in vetting immigration requests was condemned by advocacy groups as a threat to naturalized citizens for its secretive algorithmic decision-making, reliance on poor quality data and unknown sources, and alleged discrimination of immigrants using biometric and sensitive information.
Kronos Scheduling Algorithm Allegedly Caused Financial Issues for Starbucks Employees
Kronos’s scheduling algorithm and its use by Starbucks managers allegedly negatively impacted financial and scheduling stability for Starbucks employees, which disadvantaged wage workers.
Tech Companies Reportedly Influenced Gig Workers' Behaviors Using Algorithms to Vary Pay for Same Amount of Work
Amazon and Uber were alleged in a multiyear ethnographic study using algorithmic systems based on gig workers' data to vary pay, such as by offering them lower wages for the same amount of work.
Coffee Meets Bagel’s Algorithm Reported by Users Disproportionately Showing Them Matches of Their Own Ethnicities Despite Selecting “No Preference”
Users selecting “no preference” were shown by Coffee Meets Bagels’s matching algorithm more potential matches with the same ethnicity, which was acknowledged and justified by its founder as a means to maximize connection rate without sufficient user information.
New York City Reportedly Released Disputed Value-Added Teacher Ratings for Thousands of Teachers
After a legal dispute over public-records requests, the New York City Department of Education released Teacher Data Reports that used a value-added statistical model to rate thousands of public-school teachers based on estimated student test-score growth. News outlets published the ratings in February 2012, and teachers, unions, and analysts disputed the scores as imprecise and potentially misleading, citing large error margins, limited samples, and apparent roster-linkage errors.
Major Universities Are Reportedly Using Race as a 'High Impact Predictor' of Student Success
Several major universities are reportedly using a tool that uses race as one factor to predict student success.
Apple Tweaked App Store Ranking Algorithms, Allegedly Resulted in Demotion of Local Apps in China
Following Apple’s changes in ranking algorithm in its iTunes App Store, apps by allegedly reputable companies and local startups in China experienced significant drops in ranking order.
2010 Market Flash Crash
A modified algorithm was able to cause dramatic price volatility and disrupted trading in the US stock exchange.