AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
244 incidents
Tinder's Personalized Pricing Algorithm Found to Offer Higher Prices for Older Users
Tinder’s personalized pricing was found by Consumers International to consider age as a major determinant of pricing, and could be considered a direct discrimination based on age, according to anti-discrimination law experts.
UK Visa Streamline Algorithm Allegedly Discriminated Based on Nationality
UK Home Office's algorithm to assess visa application risks explicitly considered nationality, allegedly caused candidates to face more scrutiny and discrimination.
UK Home Office's Sham Marriage Detection Algorithm Reportedly Flagged Certain Nationalities Disproportionately
UK Home Office's opaque algorithm to detect sham marriages flagged some nationalities for investigation more than others, raising fears surrounding discrimination based on nationality and age.
US DHS’s Opaque Vetting Software Allegedly Relied on Poor-Quality Data and Discriminated against Immigrants
US Citizenship and Immigration Services (USCIS)’s ATLAS software used in vetting immigration requests was condemned by advocacy groups as a threat to naturalized citizens for its secretive algorithmic decision-making, reliance on poor quality data and unknown sources, and alleged discrimination of immigrants using biometric and sensitive information.
Tech Companies Reportedly Influenced Gig Workers' Behaviors Using Algorithms to Vary Pay for Same Amount of Work
Amazon and Uber were alleged in a multiyear ethnographic study using algorithmic systems based on gig workers' data to vary pay, such as by offering them lower wages for the same amount of work.
Facebook's Auto-Generated Targeting Ad Categories Contained Anti-Semitic Options
Facebook's automated advertising categories generated using users' declared interests contained anti-Semitic categories such as "Jew hater" and "How to burn Jews" which were listed as fields of study.
Coffee Meets Bagel’s Algorithm Reported by Users Disproportionately Showing Them Matches of Their Own Ethnicities Despite Selecting “No Preference”
Users selecting “no preference” were shown by Coffee Meets Bagels’s matching algorithm more potential matches with the same ethnicity, which was acknowledged and justified by its founder as a means to maximize connection rate without sufficient user information.
Sexist and Racist Google Adsense Advertisements
Advertisements chosen by Google Adsense are reported as producing sexist and racist results.
UT Austin's GRADE Algorithm Reportedly Reduced Review of Lower-Scored PhD Applicants Amid Bias Concerns
From the 2013 through 2019 admissions cycles, UT Austin's Department of Computer Science used GRADE, a statistical machine-learning system trained on past admissions decisions, to score and organize PhD applications. Critics said the system could reproduce historical admissions inequities and reduce attention to lower-scored applicants, while UT Austin said human reviewers still evaluated each file and later discontinued the tool.
Chicago Police's Strategic Subject List Reportedly Biased Along Racial Lines
The Chicago Police Department reportedly used person-based predictive policing models, including the Strategic Subject List and Crime and Victimization Risk Model, to assign violence-risk scores to hundreds of thousands of residents. Official and media reports alleged that the program gave police a mathematically fragile basis for treating heavily policed residents as future victims or suspects, with Black Chicago residents especially exposed to the consequences of that scoring.
Police Reportedly Deployed ShotSpotter Sensors Disproportionately in Neighborhoods of Color
Police departments disproportionately placed ShotSpotter sensors in black and brown neighborhoods, which is denounced by communities for allegedly creating dangerous situations, such as one involving in Adam Toledo's death.
Google Instant's Allegedly 'Anti-Semitic' Results Lead To Lawsuit In France
The organizations SOS Racisme, Union of Jewish Students of France, Movement Against Racism and for Friendship Among Peoples are suing Google due to its autocomplete software suggesting "jewish" when the names of certain public figures were searched on the platform.
Major Universities Are Reportedly Using Race as a 'High Impact Predictor' of Student Success
Several major universities are reportedly using a tool that uses race as one factor to predict student success.
IRS Audited Black Taxpayers More Frequently Reportedly Due to Algorithm
The IRS was auditing Black taxpayers more frequently than other groups allegedly due to the design of their algorithms, focusing on easier-to-conduct audits which inadvertently correlated with the group's pattern of tax filing errors.
Amazon Censors Gay Books
Amazon's book store "cataloging error" led to books containing gay and lesbian themes to lose their sales ranking, therefore losing visibility on the sales platform.