The Unheard Alternative: Contrastive Explanations for Speech-to-Text Models

Contrastive explanations reveal hidden biases in speech AI: The paper demonstrates that by comparing why a speech-to-text model chose one output over another (e.g., assigning masculine vs. feminine gender), researchers can identify which specific audio features drive these decisions—a capability previously unavailable for these systems.

Gender bias in speech translation is traceable to specific audio patterns: Through analysis of speech translation models, the researchers found that certain acoustic features in the input audio influence gender assignment, suggesting that biases in these systems aren't random but stem from identifiable patterns the model has learned.

Current explanation methods fall short for generative speech models: Standard explanations for AI systems don't adequately address how speech-to-text models work. This paper addresses that gap by introducing the first method specifically designed to provide contrastive explanations for these generative models.

Better explanations enable better accountability: By making it possible to understand why a speech-to-text system made a particular choice between alternatives, this work provides a foundation for auditing these models for fairness issues and holding developers accountable for problematic outputs.

Explainability is essential for high-stakes applications: Since speech-to-text systems are increasingly used in real-world contexts like translation and transcription, understanding their decision-making processes becomes crucial for identifying and mitigating potential harms before deployment.