A Pragmatic Way to Measure Chain-of-Thought Monitorability

Chain-of-Thought reasoning can be monitored for safety, but this capability needs active preservation. The paper warns that shifts in how AI models are trained or designed could undermine our ability to follow and verify their reasoning processes, making it important to measure and track this "monitorability" as a deliberate practice.

Two measurable components of monitorability are legibility and coverage. Legibility measures whether humans can actually follow a model's reasoning step-by-step, while coverage measures whether the reasoning shown contains everything needed for a human to independently reach the same conclusion—both are essential for meaningful oversight.

Current frontier AI models show high monitorability on challenging tasks. When tested on difficult benchmarks, several state-of-the-art models demonstrated strong legibility and coverage in their reasoning chains, suggesting that transparent reasoning is currently achievable at scale.

The paper provides a practical tool for developers to track monitorability over time. By sharing an "autorater prompt" that any capable language model can use to evaluate legibility and coverage, the authors enable organizations to monitor how their design decisions affect whether AI reasoning remains interpretable—supporting proactive rather than reactive safety measures.

This measurement approach complements but doesn't replace adversarial testing. The authors emphasize that measuring default monitorability should work alongside stress-testing designed to catch cases where models might deliberately hide or obscure their reasoning, recognizing that measurement alone is insufficient for robust AI safety.