A Formal Approach to Explainability

Explanations require formal definitions: The paper establishes that explanations cannot be treated as informal concepts—they need precise mathematical definitions to ensure consistency and reliability across different AI systems and use cases.

Explanations have verifiable properties: The research identifies specific desired properties that explanation-generating functions should satisfy, allowing developers to formally test whether their explanation methods actually meet stated goals rather than relying on intuition alone.

Consistency can be verified across model layers: A key finding shows that if a model layer's internal activations align with an explanation, all subsequent layers will also be consistent with that explanation—this provides a practical way to validate explanations throughout a neural network's architecture.

Explanations can be combined systematically: The paper demonstrates that explanations can be meaningfully combined through intersection and union operations, suggesting that multiple explanation methods could be integrated to provide more comprehensive understanding of model decisions.

Formal rigor improves trustworthiness: By grounding explanation methods in mathematical formalism rather than heuristics, this approach helps ensure that explanations are reproducible, auditable, and suitable for high-stakes applications where stakeholders need confidence in how AI systems reach their conclusions.