Evaluating Explainability in Safety-Critical ATR Systems: Limitations of Post-Hoc Methods and Paths Toward Robust XAI

Post-hoc explanation methods have critical safety limitations: The paper identifies systematic failures in widely-used XAI techniques (saliency maps, attention visualizations, surrogate models) when applied to safety-critical systems, meaning these popular tools may be insufficient for high-stakes decisions like military target recognition.

Visual explanations can be deceptively convincing but unreliable: The research documents a specific failure mode called "overtrust induced by visually convincing outputs," where explanations look plausible to human reviewers but don't actually reflect how the model makes decisions—a particular concern when non-technical personnel must validate AI systems.

Explanations are vulnerable to manipulation and instability: The paper demonstrates that explanation methods can produce spurious (false) explanations and become unstable when the input data is slightly altered, meaning adversaries could potentially game the system or minor data variations could undermine confidence in the same model's decisions.

Current XAI approaches fail key validation requirements: Across four critical dimensions—interpretability, robustness, vulnerability to manipulation, and suitability for verification—existing explanation methods show systematic gaps that prevent them from meeting the assurance standards needed for safety-critical deployment in defense applications.

Future safety-critical AI requires fundamentally different approaches: Rather than adding explanations after model training, the paper calls for "causally grounded" and "physically informed" methods built into systems from the start, suggesting that responsible AI in high-stakes domains requires rethinking how we build and validate models, not just how we explain them.