Towards Compositional Interpretability for XAI

Interpretability requires understanding model structure, not just outputs: The paper argues that true interpretability comes from analyzing how a model is built and how its components connect, rather than just explaining individual predictions. This compositional approach applies across different types of AI systems—from simple rule-based models to complex neural networks and transformers.

Current "black-box" AI models lack transparency in high-stakes decisions: The research highlights that most modern AI systems used in finance, legal, and healthcare sectors cannot be easily understood, creating significant risks. The paper proposes that examining a model's internal structure through formal diagrams can reveal why it makes decisions.

Some AI models are inherently more interpretable than others: The study identifies that certain models—including causal models, rule-based systems, and conceptual space models—have built-in transparency advantages. Understanding why these models are clearer (through their compositional structure) helps identify which AI systems are safer for critical applications.

A unified framework can compare interpretability across different AI types: Rather than treating neural networks, transformers, and causal models as completely separate, the paper uses category theory to analyze them on common grounds, making it easier to evaluate which approaches offer better explainability for specific use cases.

Compositional interpretability offers practical benefits beyond explanation: Beyond just understanding decisions, the paper suggests that models built with compositional structure may enable additional beneficial capabilities, though the abstract cuts off before fully detailing these advantages.