Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models

Foundation models can help AI systems escape local optima: The paper demonstrates that using large language models' code-generation and knowledge capabilities allows self-play algorithms to discover diverse strategies rather than getting stuck in repetitive, locally optimal behaviors—a longstanding limitation of traditional self-play approaches.

Diversity and quality aren't mutually exclusive: The Quality-Diversity Self-Play (QDSP) variant successfully combines high-performing strategies with diverse solutions, suggesting that responsible AI development can pursue both effectiveness and behavioral variety simultaneously, rather than treating them as trade-offs.

Foundation models discovered novel solution types: In the Car Tag experiment, FMSP discovered reinforcement learning, tree search, and heuristic-based methods autonomously, indicating that foundation models can generate creative problem-solving approaches beyond human-designed strategies.

This approach has direct safety applications: The paper tests FMSP on "Gandalf," an AI safety simulation where attackers attempt to jailbreak language model defenses, suggesting the method could help identify and defend against diverse adversarial strategies—relevant for building more robust AI systems.

Human-designed strategies can be surpassed: Both FMSP variants outperformed strong human-designed strategies in their experiments, raising important questions about how AI systems should be evaluated and whether human oversight remains sufficient as AI capabilities expand.