A Close Look at Decomposition-based XAI-Methods for Transformer Language Models

Decomposition-based explanation methods (ALTI-Logit and LRP) directly trace how a language model's prediction is built through its network layers, making them theoretically promising for understanding transformer decision-making, though their practical performance had not been rigorously compared until this study.

When tested on a concrete task (subject-verb agreement), different attribution methods produced notably different results, highlighting that the choice of explanation method significantly impacts what insights researchers and practitioners draw about how language models work—suggesting caution in relying on any single method.

The researchers created and released a benchmark dataset with ground truth annotations for evaluating explanation methods, addressing a critical gap in responsible AI research where explanation techniques are often evaluated without clear standards, enabling more rigorous future comparisons.

Both gradient-based and decomposition-based methods were tested across multiple model architectures (BERT, GPT-2, and LLaMA-3), revealing that explanation method performance may vary depending on the specific model, which is important for practitioners choosing tools to audit different AI systems.

The study emphasizes the need for careful, quantitative evaluation of XAI methods rather than relying on intuitive plausibility, supporting the responsible AI principle that transparency tools themselves require validation to ensure they accurately represent model behavior rather than creating false confidence in our understanding.