From Explainable to Explained AI: Ideas for Falsifying and Quantifying Explanations

Explanation tools alone aren't enough: The paper distinguishes between "explainable AI" (having tools that show what a model looks at) and "explained AI" (actually understanding why those features matter). Techniques like GradCAM can identify which parts of an image influence a model's decision, but this doesn't automatically constitute a valid explanation.

AI systems can rely on misleading patterns: The research highlights a critical risk in medical AI—models may depend on "spurious features" (irrelevant patterns that happen to correlate with outcomes) rather than genuine biological indicators. This can cause the system to fail for certain patient groups or provide false confidence in incorrect diagnoses.

Explanations should be testable and measurable: The authors propose a practical system where clinicians can actively test whether an explanation is accurate by running experiments within an AI-integrated slide viewer, and then quantify how well competing explanations actually predict outcomes. This moves beyond subjective interpretation.

Vision-language models can validate explanations: By using general-purpose AI models trained on images and text, the system can objectively assess whether an explanation's claimed features are genuinely predictive, offering a scalable way to verify explanations across different medical imaging contexts.

This approach enables safer clinical deployment: For medical AI systems to be trustworthy in hospitals, clinicians need confidence that explanations are accurate and that models aren't relying on hidden biases. This framework provides a practical pathway to achieve that confidence before clinical integration.