Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns

Transformers need better explanations for high-stakes decisions: The paper highlights that transformer models are increasingly used in critical domains like healthcare, legal systems, and finance, where understanding why a model made a decision is essential for trust and accountability—yet current methods for explaining these models remain limited.

Attention weights alone aren't enough: While transformer attention mechanisms have been used to interpret models, the paper finds that existing approaches rely on manually designed rules for combining attention information, suggesting that simple attention visualization doesn't automatically provide reliable explanations of model behavior.

A new automated approach learns better explanations: ExpNet, the method introduced in this paper, automatically discovers which attention patterns are most important for explaining decisions, rather than using predetermined rules—potentially offering more accurate and flexible explanations across different tasks and applications.

Computational efficiency matters for responsible AI: The paper contrasts ExpNet with model-agnostic methods like LIME and SHAP, which require repeatedly testing the model with modified inputs and are computationally expensive; a more efficient explanation method could make interpretability more practical for real-world deployment in regulated industries.

Explainability methods need rigorous evaluation: By benchmarking ExpNet against multiple explanation approaches across different methodological families, the paper emphasizes that responsible AI requires systematic testing of explanation methods—not just assuming that one approach works universally across all tasks and domains.