What's meant by explainable model: A Scoping Review

Most "explainable" AI systems lack proper evaluation: The paper found that 81% of application papers claiming to use explainable AI models did not actually evaluate whether their explanation methods worked, suggesting the term "explainable" is often used without rigorous validation.

Adding an explanation tool doesn't automatically make AI trustworthy: The research reveals that simply applying a post-hoc explanation method (tools added after a model is built) to an AI system is insufficient to call it "explainable"—these methods need to be tested and validated for the specific application.

Explanation quality depends heavily on context: The paper emphasizes that explanations are "application- and domain-specific," meaning an explanation method that works well for medical diagnosis may not work for loan approval decisions, requiring separate evaluation for each use case.

Popular explanation methods show inconsistent performance: Feature attribution methods—commonly used techniques to explain which data inputs influenced an AI decision—"vary substantially" in their effectiveness, indicating they are not a universal solution to the explainability problem.

Responsible AI requires evaluation, not just tools: For organizations deploying AI systems, this research underscores that claiming a model is "explainable" requires demonstrating that explanation methods actually provide meaningful, accurate information for that specific application.