Attention vs non-attention for a Shapley-based explanation method

Explainability methods designed for computer vision may not work well for language AI: The paper highlights that explanation methods are often developed and tested primarily in computer vision, but their effectiveness for natural language processing (NLP) systems—which process language differently—remains largely unexplored, suggesting a gap in how we validate AI transparency tools across different domains.

Shapley-based explanations can now work with modern attention-based language models: The researchers successfully extended Contextual Decomposition (CD), a Shapley-based explanation method, to work with attention mechanisms—the core technology behind current large language models—providing a new tool for understanding how these systems make decisions.

Attention and non-attention models process language differently despite similar outputs: When analyzing how English and Dutch models handle subject-verb relationships across different sentence structures, the study found that attention-based and non-attention models showed "consistent differences" in their internal processing, even when producing similar results—meaning the same answer can come from different reasoning paths.

Language structure matters for model transparency: By testing multiple syntactic structures across two languages, the research demonstrates that responsible AI evaluation should account for linguistic complexity and cross-language variation, not just overall accuracy metrics.