Accountable partial understanding: securing the epistemic attack surface of agentic systems

AI systems create a unique "understanding gap" problem: Unlike traditional software where unexplained behavior is accepted as risk, AI agents that make decisions, generate explanations, and authorize actions make partial human understanding more dangerous because authority is delegated to systems humans don't fully comprehend.

Four specific attack categories threaten AI accountability: Attackers can exploit gaps in (1) understanding what the AI was supposed to do, (2) trusting the evidence the AI produces, (3) knowing the limits of the AI's authority, and (4) supervisors' ability to actually oversee the system—each requiring different defensive strategies.

"Accountable Partial Understanding" requires six concrete conditions: Organizations deploying AI agents must establish bounded scope with explicit assumptions, inspectable evidence trails, clear responsibility assignment, visible documentation of unknowns, actionable oversight mechanisms, and ongoing maintenance—not just deploy systems and hope for the best.

Current approaches to AI safety are insufficient: The paper distinguishes APU from explainable AI, formal verification, and human-in-the-loop oversight, suggesting these existing methods alone don't adequately address the epistemic attack surface created by delegated AI authority.

Design must prioritize intervention and transparency: Rather than trying to make AI fully understandable, systems should be built with bounded delegation limits, decoupled evidence that can be independently verified, tracked assumptions that trigger alerts when invalid, and preserved human ability to intervene.