Explaining AI Agents Through Execution Traces

Execution traces provide a universal audit trail: The framework works across different AI agent architectures and environments by analyzing what agents actually did (their execution traces) rather than their internal mechanisms, making it applicable to diverse real-world deployments where agents interact with external tools.

Traditional explainability methods are insufficient for complex agents: The paper identifies that standard XAI approaches fail to capture the "process-level transparency" needed for multi-step decision-making systems, highlighting a critical gap in current oversight capabilities for deployed AI agents with limited human supervision.

The framework detects problematic reasoning patterns: Beyond explaining actions, the system reliably identifies "unsupported claims, unjustified actions, and evidence gaps," providing auditors with concrete tools to spot when agents make decisions without adequate justification—a key requirement for responsible deployment.

Natural language explanations grounded in observable behavior improve accountability: By converting execution traces into structured reports and faithful natural-language explanations, the framework makes agent decision-making more interpretable to non-technical stakeholders, supporting better human oversight and governance.

Performance outpaces simpler alternatives: Human and automated evaluations show the framework produces higher-quality explanations than naive LLM-generated explanations, suggesting that purpose-built approaches for agent transparency are more reliable than generic explanation methods.