Multi-Agent AI Safety as an Institutional Design Problem

Institutional design rules matter for AI safety: The paper demonstrates that how AI systems are structured—including delegation rules, information flow, and authority checks—significantly impacts whether they behave safely. A detailed constitutional prompt achieved zero violations across 384 test episodes, showing that explicit governance frameworks can be effective safeguards.

Different safety mechanisms have different failure modes: The study compared three types of guards (constitutional prompts, provenance-aware executable guards, and local-state guards) and found they fail in different ways. For example, the local-state guard specifically failed when policies changed while underlying authority remained the same, suggesting that no single mechanism catches all problems.

Transparency about constraints can improve compliance: When the system revealed numerical values of resource caps (rather than keeping them hidden), it changed how AI agents behaved in resource-allocation scenarios. This suggests that making governance rules visible to AI systems, rather than obscuring them, may support safer outcomes.

Some safety mechanisms prevent violations while others allow recovery: A provenance-aware guard blocked prohibited attempts in 51 out of 384 episodes, but 44 of those episodes later completed safely through alternative paths. This indicates that safety can work through both prevention and allowing systems to find compliant workarounds, rather than just blocking actions entirely.

Rigorous testing reveals specific vulnerabilities: The study identified particular scenarios—called "laundering scenarios"—where certain guards failed (22 violations in 96 episodes) while others succeeded (0 violations), demonstrating the importance of targeted diagnostic testing to find edge cases in multi-agent AI systems.