The Concept of Criticality in AI Safety

Critical actions require human oversight: The paper identifies that not all AI actions need human approval—only "critical actions" (those with potential to cause harm) require operator permission, making human monitoring practical rather than requiring constant surveillance of every decision.

Human-in-the-loop monitoring can be efficient: Rather than requiring an operator to dedicate all their attention to monitoring an AI agent, this approach allows humans to focus on other tasks while still maintaining safety oversight, addressing a major practical barrier to deploying monitored AI systems.

Measuring action criticality is key: The paper emphasizes that building a reliable model to identify which actions are critical is essential to the approach—the system must accurately distinguish between routine, low-risk actions and potentially harmful ones to function effectively.

Operator feedback improves AI performance: Beyond just preventing harm, the paper suggests that human feedback on critical actions can be used to make the AI agent "smarter," potentially improving both safety and capability over time through a learning process.

This approach balances safety with practicality: The paper addresses a fundamental tension in AI safety—that perfect safety through constant human monitoring is unrealistic, proposing instead a middle ground that maintains meaningful human control while remaining operationally feasible.