What do we need to build explainable AI systems for the medical domain?

Legal requirements are driving explainable AI adoption in medicine: The paper highlights that the European General Data Protection Regulation (GDPR), which entered into force on May 25, 2018, requires that AI decisions be "re-traceable on demand," making it legally difficult to deploy unexplainable "black-box" AI systems in medical applications without the ability to explain results when requested.

Medical AI systems need transparency despite impressive performance: While deep learning models have achieved remarkable results in medical applications and even exceeded human performance in some visual tasks, the paper emphasizes that their "black-box" nature—lacking explicit explanatory structures—creates a fundamental problem that must be addressed before widespread medical deployment.

Explainability is distinct from understanding the math: The paper clarifies an important distinction: even when researchers understand the underlying mathematical principles of AI models, this technical knowledge doesn't automatically make the systems explainable to doctors, patients, or regulators who need to understand why a specific medical decision was made.

Medical domain requires special considerations: The authors emphasize that medicine is "a very special domain" requiring tailored approaches to explainable AI, suggesting that solutions developed for other AI applications (like autonomous driving or game-playing) may not directly transfer to healthcare contexts where patient safety and trust are paramount.