Agent Safety Should Be a Runtime Contract

Training alone is insufficient for agent safety: The paper argues that current approaches relying on training methods like RLHF and Constitutional AI are "structurally insufficient" for autonomous agents that take real-world actions (executing code, modifying files, sending messages). Safety must be enforced at runtime through system-level controls, not just model behavior.

A two-part runtime safety contract is needed: Effective agent safety requires both a "preventive face" (sandboxes, permission gates, output filters) that blocks dangerous actions before they occur, and an "evidential face" that requires verifiable proof actions were completed correctly (test runs, log captures, file diffs), rather than trusting the agent's claims.

Current research focus is misaligned with deployment reality: Analysis of 28,560 papers from top AI conferences (2023-2025) shows an 8-12x imbalance favoring training-time safety research over deployment-time safety—despite real-world incidents demonstrating that runtime enforcement is critical for autonomous systems.

Real incidents reveal training-based safety gaps: A survey of 52 documented AI-agent and LLM safety incidents, combined with a "false-completion audit" of 31 cases, demonstrates that models trained to be safe still fail to execute safely in practice, supporting the need for external runtime controls.

Proven safety models exist in other fields: Computer security and experimental sciences have successfully enforced safety through runtime contracts combining preventive and evidential elements—agentic AI should adopt similar proven approaches rather than relying solely on training.