From Features to Actions: Explainability in Traditional and Agentic AI Systems

Traditional explainability methods don't work for AI agents: The paper finds that explanation techniques designed for single predictions (like identifying which input features matter most) are unreliable when applied to AI systems that make sequences of decisions over time. While these methods achieved 86% consistency in static settings, they failed to diagnose why autonomous AI agents made mistakes.

State tracking problems are a major cause of agent failures: When researchers examined AI agents that failed at multi-step tasks, they discovered that inconsistent state tracking (the agent losing track of information it should remember) was 2.7 times more common in failed runs and reduced success rates by 49%—suggesting this is a critical vulnerability to monitor.

We need new explanation approaches for autonomous AI systems: The paper argues that as AI systems become more autonomous and make decisions over multiple steps, we need fundamentally different explanation methods focused on understanding entire action sequences ("trajectories") rather than individual decisions, shifting from "why did the model predict X?" to "why did the agent fail at this task?"

Trace-based diagnostics outperform traditional methods for agents: The researchers found that examining the step-by-step execution trace of an agent's actions—combined with structured evaluation rubrics—successfully identified where and why autonomous systems broke down, offering a more practical alternative to traditional feature attribution methods.