LIG: Layer-wise Integrated Gradients for Within-Layer Flow Analysis in Transformers

Transformers' internal decision-making can now be traced at finer granularity: LIG enables researchers to track how information flows through specific components (attention heads and MLPs) within each layer, moving beyond black-box analysis to understand which parts of the model contribute to specific outputs.

A new method for explaining AI systems without modifying the model: Unlike some interpretability approaches, LIG works as a diagnostic tool on existing trained models like BERT without requiring retraining or custom modifications, making it practical for analyzing AI systems already in use.

Information flow within layers shows consistent patterns: The research found that certain baseline configurations (using the target token's embedding for attention and zero values for MLP layers) best preserved internal consistency, suggesting Transformer layers have identifiable, analyzable structures that could inform future model design.

Explainability methods can be mathematically rigorous: By extending Integrated Gradients—a technique with formal completeness guarantees—to analyze token-to-token contributions, this work demonstrates that interpretability tools can maintain mathematical soundness while becoming more granular, supporting trustworthiness claims about AI explanations.

Better interpretability supports responsible AI deployment: Understanding how Transformers process information at the module level provides a foundation for detecting biases, errors, or unexpected behaviors before deployment, which is essential for building more accountable AI systems.