Explanation Beyond Intuition: A Testable Criterion for Inherent Explainability

Explainability needs a testable definition, not just intuition: The paper addresses a critical gap in AI regulation by proposing the first formally testable criterion for "inherent explainability" rather than relying on subjective judgments. This provides regulators and organizations with a rigorous framework for compliance, moving beyond the current "we know it when we see it" approach.

Model size and complexity don't determine explainability: The research demonstrates that a large regression model may fail explainability tests while a sparse neural network could pass them. This challenges common assumptions that simpler models are automatically more explainable and suggests that model structure matters more than model size.

Explainability requires verified evidence, not just capability: The paper distinguishes between "explainable" (a model that allows for explanation) and "explained" (one with verified explanation). This distinction is important for responsible AI deployment—having the potential to explain a model is insufficient; organizations must actually provide and verify those explanations.

Real-world medical AI can meet rigorous explainability standards: The authors provide a complete explanation of PREDICT, a cardiovascular disease risk model actively used in New Zealand clinics, demonstrating that inherent explainability is achievable in high-stakes applications. This provides a concrete example that rigorous explainability standards are practical, not theoretical.

A flexible framework can serve regulatory needs: The proposed criterion uses graph theory and hypothesis-evidence structures that allow multiple explanation methods while maintaining rigor, offering regulators a tool that is both flexible enough for diverse AI systems and strict enough to ensure genuine accountability.