Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers

Current explanation methods for AI transformers are fundamentally flawed: The paper formally proves that traditional feature attribution methods—which rely on linear or additive models to explain decisions—are structurally incompatible with how transformers actually work, meaning explanations from these methods may be misleading or unreliable.

This incompatibility has widespread implications: Since transformers power most modern natural language processing applications (chatbots, translation tools, content moderation systems), the flawed explanations affect a vast range of AI systems currently deployed in high-stakes domains.

A new explanation method (SLALOM) better matches transformer architecture: The researchers developed an alternative approach specifically designed to align with how transformers function, demonstrating it can provide more accurate explanations while requiring less computational resources than existing methods.

Better explanations are essential for responsible AI deployment: The paper highlights why understanding how transformers make decisions matters—without accurate explanations, organizations cannot properly audit these systems for bias, errors, or harmful outputs before deployment.

This work is open-source and actionable: The authors released code for SLALOM publicly, enabling practitioners to implement more reliable explanations for transformer-based systems rather than relying on methods known to be theoretically incompatible with the underlying architecture.