Reportedly Unsafe Deployment of Llama.cpp Reveals Interactive AI-Generated CSAM Roleplay Prompts
Description
A study by UpGuard reports that misconfigured llama.cpp servers publicly exposed user prompts, including hundreds of interactive roleplay scenarios. Some prompts explicitly described fictional sexual abuse of children aged 7–12. While no real children were involved, the findings demonstrate how open-source LLMs can be exploited to generate AI-enabled child sexual abuse material (CSAM).
Actors
Alleged Deployers
- users-of-llama.cpp-servers
Alleged Developers
- meta
- users-of-llama.cpp-servers
Harmed Parties
- general-public
- users-of-llama.cpp-servers
Classification
Risk Domain
Privacy & SecurityRisk Subdomain
2.1. Compromise of privacy by obtaining, leaking or correctly inferring sensitive information
MIT Risk Profile
Topics
Source
Related Incidents (Privacy & Security)
View all →Research Papers on This Topic
View all →StarCoder: may the source be with you!
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
LineVD: Statement-level Vulnerability Detection using Graph Neural Networks
TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems
Mitigation Tools
Tools related to this risk domain, via shared responsible AI topics.
Open Source
View all →flair
flairNLP/flair
A very simple framework for state-of-the-art Natural Language Processing (NLP)
⭐ 14,353
PySyft
OpenMined/PySyft
Perform data science on data that remains in someone else's server
⭐ 10,041
FATE
FederatedAI/FATE
An Industrial Grade Federated Learning Framework
⭐ 6,100
adversarial-robustness-toolbox