The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

Self-play red teaming has a fundamental flaw: When the same AI model plays both attacker and defender roles, the training dynamics can collapse into "self-consistency," where the attacker stops genuinely challenging the defender. This means the safety improvements may be illusory rather than robust.

Separate roles improve safety outcomes: The researchers' new method (Anchored Bipolicy Self-Play) uses distinct, specialized versions of the model for attack and defense roles, which maintains genuine adversarial pressure. This approach showed consistent safety improvements compared to standard self-play methods across multiple model sizes (3B to 14B parameters).

Efficiency gains without sacrificing safety: The new approach achieves up to 100x greater parameter efficiency than traditional fine-tuning methods, meaning organizations can improve AI safety more cost-effectively without requiring massive computational resources or model retraining.

Current safety guarantees may be overstated: The paper reveals that even when self-play reaches a theoretical "Nash equilibrium" (a stable game outcome), the resulting safe behaviors can range from trivial "always refuse" responses to more nuanced safety—suggesting that current mathematical guarantees don't necessarily translate to practical, useful AI safety in real-world applications.