SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation

Speech AI systems now have better explainability tools: SPES addresses a significant gap in AI interpretability by providing the first feature attribution method designed specifically for speech-to-text generation tasks, moving beyond previous techniques that only worked for text classification or didn't account for how these models generate output word-by-word.

Explanations can be both technically sound and understandable to humans: The paper demonstrates that SPES generates explanations that are "faithful and plausible to humans," meaning the system can show which parts of an audio recording influenced each word in the transcription in ways that make intuitive sense to non-technical users.

Context matters in understanding AI decisions: Unlike simpler explanation methods, SPES accounts for the fact that speech recognition models make decisions based on both the incoming audio and previously generated text, providing more complete and accurate explanations of how these systems work.

Responsible AI for speech technology is catching up to text-based AI: This work highlights how explainability research in speech and audio has lagged behind similar efforts in natural language processing, and SPES helps close that gap—important as speech-based AI systems become increasingly common in real-world applications.