Evaluating the Faithfulness of Saliency-based Explanations for Deep Learning Models for Temporal Colour Constancy

Attention mechanisms don't reliably explain how AI models make decisions: The study found that attention-based saliency methods—commonly used to show which parts of a video a model "focuses on"—failed faithfulness tests, meaning they don't accurately represent the model's actual decision-making process, even though they're widely promoted as interpretability tools.

Different saliency methods have different levels of trustworthiness: While attention failed the faithfulness tests, the researchers found that "confidence" (another type of visual saliency measure) successfully passed, suggesting that not all explanation methods are equally reliable for understanding model behavior.

Saliency faithfulness problems extend beyond language to video analysis: Previous concerns about saliency methods were mainly documented in natural language processing; this paper demonstrates the same faithfulness issues occur in video processing tasks like color constancy, indicating the problem is broader than previously known.

Current explanation methods may give false confidence in AI systems: The findings suggest that using unfaithful saliency explanations could mislead developers and users into believing they understand how a model works when they actually don't, potentially hindering debugging and improvement efforts.

Rigorous testing of AI explanations is essential before deployment: The researchers adapted and refined faithfulness tests from NLP literature, highlighting that explanation methods need systematic evaluation before being relied upon for critical applications or regulatory compliance.