Opening the Black Box: Analyzing Attention Weights and Hidden States in Pre-trained Language Models for Non-language Tasks

Language models solve structured math problems similarly to humans: When applied to hierarchical arithmetic problems, the pre-trained language model demonstrated "moderately structured" problem-solving approaches comparable to human thought processes, suggesting these models may develop interpretable reasoning strategies rather than purely opaque pattern matching.

Not all layers are equally important for fine-tuning: The research found that layer 10—rather than the final layer—was optimal for efficient model adaptation, requiring fewer parameters to fine-tune while maintaining performance. This challenges common assumptions about which parts of language models are most critical for task-specific learning.

Attention patterns reveal model capabilities: By analyzing attention weights layer by layer, researchers could predict that the model would generalize to longer sequences than it was trained on—a prediction later confirmed through testing. This demonstrates that examining internal model mechanisms can provide genuine insights into model behavior before deployment.

Explainability is achievable through systematic analysis: The paper shows that "black box" language models can be meaningfully understood by examining their internal components (attention weights and hidden states) using simple, interpretable tasks, supporting the feasibility of explainable AI without sacrificing model performance.

Interpretability requires appropriate evaluation methods: The researchers combined multiple analytical approaches—attention analysis, entropy analysis, and token embedding similarity—to validate their findings, suggesting that responsible AI requires rigorous, multi-faceted investigation rather than relying on single interpretability methods.