AI Incidents
Real-world cases where AI systems caused harm, classified by risk domain and entity. Data sourced from 
AIID and classified by 
MIT AI Risk Initiative. CC BY-SA 4.0
217 incidents
Nine Network's AI Alters Lawmaker Georgie Purcell's Image Inappropriately
The Nine Network used Photoshop's Generative Expand AI tool to resize an image of lawmaker Georgie Purcell, inadvertently altering her attire to appear more revealing. This error, claimed to result from the AI's automation, led to public criticism and an apology from the network.
Chatbot for DPD Malfunctioned and Swore at Customers and Criticized Its Own Company
DPD's AI chatbot, used for customer service, appeared to malfunction following a system update, leading to inappropriate responses including swearing and criticizing the company. The incident, which became viral on social media, occurred after the chatbot was updated, prompting DPD to disable the malfunctioning AI component.
Rite Aid Facial Recognition Disproportionately Misidentified Minority Shoppers as Shoplifters
Rite Aid used facial recognition technology from October 2012 to July 2020, allegedly leading to disproportionate misidentifications of women, Black, Latino, and Asian shoppers as "likely" shoplifters. The FTC settlement prohibits Rite Aid from using this technology in stores for five years.
Navy Federal Credit Union Faces Allegations of Racial Bias in Mortgage Approvals
Navy Federal Credit Union, serving military members and veterans, faced allegations of racial bias in its mortgage approval process, which relies on automated underwriting technology. In 2022, data revealed significant disparities in loan approvals, with over 50% of Black applicants denied, compared to higher approval rates for white applicants.
Microsoft AI Is Alleged to Have Generated Violent Imagery of Minorities and Public Figures
Microsoft's AI Image Creator, integrated with Bing and Windows Paint, allegedly produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.
Alleged Misuse of PicSo AI for Generating Inappropriate Content Emphasizing "Girls"
PicSo AI, which appears to be getting advertised by Meta over Instagram, is allegedly being used for generating inappropriate content with an emphasis on "girls." This raises concerns about the misuse of generative AI technologies for creating offensive and potentially sexually explicit material that could be used for nefarious and criminal purposes.
Facebook Messenger AI Stickers Generate Ethical and Content Moderation Concerns
Facebook Messenger AI stickers, a feature by Meta, allows users to generate personalized stickers via AI for use in conversations. While the feature has been praised for its creativity, it has also stirred controversy for its alleged production of inappropriate or offensive content. This has raised questions about the effectiveness of Meta's content moderation measures and the ethical responsibilities associated with AI-driven content generation.
Peppermill Casino Facial Recognition System Reportedly Misidentified Individual, Leading to Wrongful Arrest in Reno
Purported facial recognition technology deployed at Reno's Peppermill Casino reportedly misidentified Jason Killinger as an individual previously banned from the venue. Casino security reportedly detained Killinger and contacted police, leading to his arrest by the Reno PD. He was reportedly held for 11 hours, including several hours in handcuffs, before fingerprinting reportedly confirmed the misidentification. The incident resulted in alleged physical injury and civil liberties harms.
AI Photo Filter Lightens Skin, Changes Eye Color in Student's 'Professional' Image
An AI application modified an MIT student's photo to appear 'professional' by lightening her skin and changing her eye color to blue, highlighting the racial bias in the training data of the program.
Harmful Stereotyping of Non-Cisgendered People via Text-to-Image Systems
Text-to-image systems such as DALL-E are allegedly generating biased and often insulting representations of non-cisgender identities. The systems tend to generate stereotypical and sexualized images when prompted with gender identity terms like "trans," "nonbinary," or "queer," highlighting systemic issues of bias.
Alleged Gender Discrimination in Facebook Job Ads Algorithm
Facebook's ad delivery algorithm allegedly disproportionately showed job advertisements to one gender. Despite claims of non-discrimination, the algorithm's actions seem to perpetuate societal biases, which in turn could potentially limit opportunities for certain groups and hinder gender equity in the workplace.
Instagram Algorithms Allegedly Promote Accounts Facilitating Child Sex Abuse Content
An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning given Instagram's popularity among teenagers.
Racial Bias in Lung Function Diagnostic Algorithm Leads to Underdiagnosis in Black Men
A study published in JAMA Network Open reveals that racial bias built into a commonly used medical diagnostic algorithm for lung function may be leading to underdiagnoses of breathing problems in Black men. The study suggests that as many as 40% more Black male patients might have been accurately diagnosed if the software were not racially biased. The software algorithm adjusts diagnostic thresholds based on race, affecting medical treatments and interventions.
Apparent Failure to Accurately Label Primates in Image Recognition Software Due to Alleged Fear of Racial Bias
Eight years after Google Photos mislabeled images of Black individuals as "gorillas," image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.
Australian Terrorism Prediction Tool Reportedly Disparately Impacts Persons with Autism
An independent report found that the Vera-2R tool, designed to predict the risk of future terrorist activities, considered autism as a risk factor despite lacking empirical evidence to support this claim. The report called into question the tool's overall validity and reliability, stating it was "extremely poor" at accurately predicting risk. The inclusion of autism as a risk factor had potentially serious implications for the tool's use and credibility.