Intent Laundering: AI Safety Datasets Are Not What They Seem

Current AI safety datasets rely too heavily on obvious "trigger words" rather than realistic adversarial attacks. The paper finds that these datasets overuse overt negative language and sensitive phrases designed to explicitly activate safety mechanisms, which doesn't reflect how real-world attacks actually work.

Popular safety datasets may create a false sense of security. When researchers removed triggering cues while preserving the actual harmful intent behind attacks, models previously considered "reasonably safe"—including Gemini 3 Pro and Claude Sonnet 3.7/4—became unsafe, suggesting current evaluations don't measure genuine safety.

A simple technique called "intent laundering" defeats current safeguards at scale. By abstracting away obvious trigger words while keeping malicious intent intact, the researchers achieved 90-100% attack success rates across models under black-box conditions, demonstrating a critical vulnerability in how AI systems are currently protected.

There's a significant gap between how AI safety is tested and real-world risks. The paper reveals that existing datasets fail to faithfully represent actual adversarial behavior, meaning companies may be deploying models that appear safe in controlled tests but remain vulnerable to more sophisticated attacks.

Better safety evaluation methods are urgently needed. The findings suggest that current approaches to measuring AI safety are fundamentally flawed and that the industry needs more realistic, intent-focused evaluation datasets rather than those relying on surface-level linguistic cues.