Discriminating Spatial and Temporal Relevance in Deep Taylor Decompositions for Explainable Activity Recognition

Existing explanation methods designed for images don't automatically work well for videos: The paper demonstrates that deep Taylor decomposition, a technique successful for explaining image-based AI models, fails to effectively explain video processing models "as-is," highlighting that spatio-temporal (space and time combined) data requires different approaches than static images.

Motion information creates "ambiguous" explanations in current video AI interpretability: The researchers found that standard explanation techniques obscure which parts of a video's importance come from motion versus static visual content, making it unclear what the AI model actually "sees" when making decisions about activities.

Separating spatial and temporal relevance improves transparency: The proposed method breaks down AI explanations into two distinct components—what matters spatially (where in the frame) and what matters temporally (motion/change over time)—making it clearer which visual elements drive the model's activity recognition decisions.

This work addresses a gap in responsible AI for video applications: As AI systems increasingly process video data for surveillance, healthcare, and other applications, having better tools to explain why these systems make decisions is critical for accountability and identifying potential biases.

Testing on real-world action recognition shows practical relevance: The technique was validated on the UCF-101 dataset, a standard benchmark for activity recognition, suggesting applicability to real video analysis systems.