Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning

Interpretability requires alignment with human understanding, not just technical clarity: The paper argues that a representation is only truly interpretable if humans can actually understand it, shifting focus from whether AI systems can explain themselves to whether explanations match how humans think about concepts.

Current AI explanation methods use incompatible approaches: The research identifies that existing explainable AI systems acquire interpretable concepts through "mutually incompatible strategies," meaning there's no standardized way to ensure different AI systems produce comparable or reliable explanations.

A mathematical framework can bridge the gap between machines and humans: The authors propose using causal representation learning to formally model human stakeholders as "external observers," creating a principled way to align machine-learned concepts with human vocabulary and understanding.

"Alignment" between AI and human concepts is distinct from technical properties: The paper clarifies that alignment (matching human understanding) is separate from disentanglement (a technical property of AI representations), suggesting that technically sophisticated AI systems may still fail to communicate effectively with humans.

The "name transfer game" provides a practical test for interpretability: The authors use a simple conceptual framework—whether humans and machines can agree on naming concepts—to operationalize and measure whether AI explanations are truly interpretable to their intended audience.