Speech Representation Analysis based on Inter- and Intra-Model Similarities

Self-supervised speech models learn similar underlying patterns despite different training approaches: The study found that models trained with different methods (Wav2Vec2.0 using contrastive learning vs. HuBERT using predictive learning) converge to similar representation subspaces, suggesting these models independently discover comparable ways to encode speech information.

Model transparency remains limited even when representations are similar: While different models learn similar overall patterns, the research reveals they do not develop the same neuron-level concepts, meaning we still cannot easily identify which specific parts of a model are responsible for understanding particular speech features—a key challenge for responsible AI oversight.

Information is distributed across models rather than localized: The findings show that speech understanding is encoded in a distributed way across layers and attention weights rather than concentrated in individual neurons, making it harder to audit, explain, or predict how these models will behave in new situations.

Model size affects representation patterns: By comparing base and large versions of models, the research demonstrates that scaling changes how information is organized internally, which has implications for understanding whether larger models are inherently more interpretable or trustworthy.

Publicly available analysis tools can advance AI accountability: The authors released their code to enable further research, supporting the principle that transparent, reproducible analysis of AI systems is essential for responsible development and deployment.