Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features

Speech AI models are harder to explain than other AI systems: While explainability methods exist for image and text models, speech models have received little attention from XAI researchers. This paper addresses a significant gap by developing the first systematic approach to explain how speech classification models make decisions.

Two-level explanations make speech models more transparent: The researchers developed a method that explains model decisions at both the word level (showing which spoken words matter most) and the paralinguistic level (showing how tone, prosody, and background noise influence predictions). This dual approach makes explanations more interpretable for non-technical users compared to existing methods.

The explanations are validated as both accurate and human-understandable: The authors tested their approach on state-of-the-art speech models across English and Italian tasks, confirming that explanations faithfully represent how the models actually work while remaining plausible to human evaluators—a critical requirement for trustworthy AI systems.

Paralinguistic features reveal hidden decision factors: Beyond just identifying important words, the method reveals that models rely on subtle audio characteristics like prosody (rhythm and intonation) and background noise. This finding highlights that speech models may be influenced by factors beyond semantic content, which has implications for fairness and bias in speech AI applications.

This work enables better oversight of speech-based AI systems: By making speech models more interpretable, this research supports responsible AI deployment in applications like voice assistants, automated call centers, and accessibility tools where understanding model decisions is essential for accountability.