Confidence-Controlled XAI Auditing for Pedestrian Detection under Domain Shift

Explanation reliability varies significantly across different driving environments: The study found that explanations for pedestrian detection systems perform differently depending on the dataset used (PIE vs. JAAD), even when using the same detector. This means explanations that appear trustworthy in one real-world driving context may not be equally reliable in another, raising concerns for autonomous vehicle deployment across different regions or conditions.

Detection confidence strongly influences whether explanations are actually faithful: The research discovered a correlation between 0.70-0.82 between a model's confidence in its detection and the faithfulness of its explanations. This means explanations for high-confidence detections may be more reliable than those for low-confidence ones, and comparing explanations without accounting for this factor can lead to misleading conclusions about model behavior.

Standard explanation auditing methods can be unreliable without proper controls: The study shows that naive comparisons of explanation quality across domains are problematic because they don't account for detection strength. This suggests that current practices for auditing AI explanations in safety-critical applications like autonomous driving may be insufficient and need more rigorous statistical controls.

The problem affects multiple explanation methods, not just one approach: Both D-RISE and EigenCAM (different explanation techniques) showed similar issues with confidence coupling, indicating this is a fundamental challenge in how deletion-based explanations are evaluated rather than a flaw in any single method.

Confidence-controlled auditing is necessary for safety-critical AI systems: The authors recommend that future audits of perception models for autonomous vehicles should explicitly control for detection confidence when evaluating explanations, particularly when assessing performance across different real-world driving scenarios.