Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology

Popular explanation methods may be unreliable: The study found that attention-based heatmaps and gradient-based saliency maps—commonly used to interpret AI models in medical imaging—"often fail to reflect model decision mechanisms," meaning they may not accurately show what the model actually uses to make predictions.

Better alternatives exist but require more computation: Perturbation-based methods, layer-wise relevance propagation (LRP), and integrated gradients (IG) consistently outperformed simpler approaches, suggesting that more rigorous explanation techniques should be prioritized in clinical AI applications, even if they require additional computational resources.

Explanations vary significantly by model type and task: The research demonstrated that "explanation quality mostly depends on MIL model architecture and task type," indicating that there is no one-size-fits-all explanation method—validation approaches must be tailored to specific AI systems and medical prediction tasks.

Explanations can now be validated without additional expert labeling: The framework introduced enables evaluation of explanation quality "without requiring additional labels," which is significant because obtaining expert annotations in medical imaging is expensive and time-consuming, making this approach more practical for real-world deployment.

Better explanations enable genuine scientific discovery: The study showed that improved explanation methods could identify distinct strategies the model uses for predictions (such as different approaches to detecting HPV infection), suggesting that trustworthy explanations may help researchers understand disease biology rather than just validating black-box predictions.