Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

Mechanism design alone cannot guarantee safe AI cooperation: The paper formally proves that even well-designed rules and incentive structures have inherent limitations when they cannot account for all possible future scenarios. This means that relying solely on external rules to make AI agents behave cooperatively is insufficient.

Prosocial AI agents outperform purely incentive-driven ones: Agents programmed to care about others' welfare alongside their own achieve better outcomes than those motivated only by personal gain. In resource-allocation and social dilemma scenarios, prosociality led to both socially superior results and individual benefits for the agents involved.

The "incomplete contract problem" creates unavoidable welfare losses: Because real-world contracts cannot anticipate every possible situation or contingency, there will always be gaps where AI agents could exploit loopholes. No realistic mechanism can completely eliminate this problem, making it a fundamental challenge in AI safety.

Building intrinsic values into AI is necessary for scale: Rather than relying exclusively on external oversight and rule-based systems, the paper suggests that AI agents deployed at scale should be designed with built-in prosocial values—essentially making cooperation part of their core objectives rather than just a response to incentives.

This research applies directly to large language models: The experiments used LLM-powered agents in realistic multi-agent scenarios, suggesting these findings are relevant to current AI systems that are already being deployed in interactive environments.