Where and When: Space-Time Attention for Audio-Visual Explanations

AI systems can now explain decisions about videos by showing both where and when they're looking: The researchers developed a method that identifies which visual elements (the "where") and at what moments in time (the "when") an AI system uses to recognize audio-visual events, moving beyond previous explanation techniques that only worked with still images.

Multi-sensory AI requires new explanation approaches: Traditional explainability methods designed for single-modality systems (like image recognition) don't adequately capture how AI systems combine audio and visual information together, highlighting a gap in responsible AI practices for increasingly complex real-world applications.

Explanations were validated through human testing and robustness checks: The researchers didn't just claim their explanations were good—they tested them using "perturbation tests" (removing elements to see if predictions changed) and "pointing games" with human annotations, demonstrating a commitment to verifying that explanations actually reflect what the model relies on.

This work addresses a practical accountability need: As AI systems increasingly process multiple types of sensory data simultaneously (audio, video, text), being able to explain why they made decisions becomes crucial for deployment in real-world scenarios where accountability matters, such as content moderation or accessibility applications.