At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors
Description
At least 10,000 AI chatbots have allegedly been created to promote harmful behaviors, including eating disorders, self-harm, and the sexualization of minors. These chatbots, some jailbroken or custom-built, leverage APIs from OpenAI, Anthropic, and Google and are hosted on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.
Actors
Alleged Deployers
- character.ai
- spicy-chat
- chub-ai
- crushon.ai
- janitorai
- unidentified-online-communities-using-chatbots
Alleged Developers
- openai
- anthropic
Harmed Parties
- vulnerable-chatbot-users
- minors-using-chatbots
- individuals-with-eating-disorders
- individuals-struggling-with-self-harm
- adolescents
Classification
Risk Domain
Discrimination & ToxicityRisk Subdomain
1.2. Exposure to toxic content
MIT Risk Profile
Topics
Source
Related Incidents (Discrimination & Toxicity)
View all →Research Papers on This Topic
View all →Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Explaining Explanations: An Overview of Interpretability of Machine Learning
RISE: Randomized Input Sampling for Explanation of Black-box Models
Mitigation Tools
Tools related to this risk domain, via shared responsible AI topics.
Open Source
View all →AIF360
Trusted-AI/AIF360
A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.
⭐ 2,872
fairlearn
fairlearn/fairlearn
A Python package to assess and improve fairness of machine learning models.
⭐ 2,199
responsible-ai-toolbox
microsoft/responsible-ai-toolbox
Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.