Both eyes open: Vigilant Incentives help Regulatory Markets improve AI Safety

Regulatory Markets for AI could help governments keep pace with rapid AI development by setting outcome-based safety targets that companies must meet through private regulators, rather than relying on slower traditional rule-making processes.

The wrong incentive structure can backfire dangerously: The paper warns that "Bounty Incentives" (which only reward regulators for catching problems) actually discourage thorough oversight because AI companies learn to match their behavior to how much effort regulators invest, creating perverse incentives against innovation in safety testing.

"Vigilant Incentives" offer a better alternative: Governments should pay private regulators consistently, but penalize them when they fail to catch unsafe behavior they should have detected. This approach encourages regulators to continuously innovate in their evaluation methods rather than settling for minimal effort.

Incentive design is critical to AI safety outcomes: The paper demonstrates through evolutionary game theory modeling that small differences in how regulators are paid can fundamentally change whether a regulatory system actually deters reckless AI behavior or inadvertently enables it.

Private regulation requires careful government oversight: Rather than simply outsourcing AI safety to the private sector, governments must actively design the incentive structures that determine whether private regulators will genuinely protect public safety or cut corners.