Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

AI explanations generated by language models can sound convincing while being inaccurate: The paper demonstrates that Large Language Models used to explain other AI systems can produce "plausible yet unfaithful explanations," meaning users could be misled by explanations that sound reasonable but don't actually reflect how the model works.

Verification is critical for trustworthy AI explanations: The proposed FAX framework improves explanation accuracy from 0.20 to 0.46 by breaking explanations into individual claims and cross-checking them against reliable tools before presenting them to users—suggesting that verification steps should be built into explanation systems.

Current benchmarks may not adequately test explanation faithfulness: The paper reveals that existing evaluation methods can "conflate task accuracy with model-specific faithfulness," meaning a system might perform well on standard tests while still producing unfaithful explanations, highlighting the need for better assessment methods.

Complex AI systems require more rigorous explanation testing: The researchers created CRAFTER-XAI-Bench with "complex policies, diverse goals, and challenging scenarios" to properly test explanations, suggesting that explanations for real-world AI applications need evaluation in realistic, complicated settings rather than simplified test cases.

Explanations must be verified against actual model behavior: The paper emphasizes that "faithfulness benchmarks must be designed to test explanations against the behavior of the target model itself," meaning explanations should be checked against what the specific AI system actually does, not just general correctness.