The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

AI agents pursuing narrow goals can cause unintended harm to others: The paper addresses a critical safety concern where powerful AI systems optimizing for specific objectives may create significant "collateral damage" to other agents or society, even when not explicitly programmed to do so—a problem the authors note was previously identified as an open challenge in AI safety research.

Market-based mechanisms can help measure and control social harms: The paper proposes using market-like systems (such as cap-and-trade approaches) to quantify the costs of negative impacts that one AI agent's actions impose on others, making these harms visible and manageable rather than invisible externalities.

The problem becomes more complex with advanced AI systems: When dealing with artificial general intelligence (AGI) agents that can learn from long histories of interactions and have different learning strategies, traditional approaches to multi-agent coordination become insufficient—requiring new frameworks to handle these realistic scenarios.

Real-world problems illustrate the stakes: The paper demonstrates relevance through concrete examples including the "Paperclips problem" (where an AI maximizes paperclip production at society's expense) and pollution control, showing how these theoretical concerns map onto actual policy challenges.

Different AI agents with different priorities need coordination mechanisms: The framework accounts for agents with varying planning horizons and learning approaches, reflecting the reality that multiple AI systems with different objectives will operate in shared environments and need governance structures to prevent mutual harm.