Can we Agree? On the Rashōmon Effect and the Reliability of Post-Hoc Explainable AI

Small datasets produce unreliable AI explanations: The study found that explanations from models trained on fewer than 128 samples showed "high variability," meaning the same model might give different explanations depending on random factors. This is critical because many organizations deploy AI systems with limited data, potentially making their explanations untrustworthy.

The Rashōmon effect reveals a fundamental problem: Multiple different machine learning models can make equally accurate predictions while disagreeing on why they make those predictions. This paper demonstrates this isn't just a theoretical concern—it directly affects whether we can extract reliable knowledge from AI systems used in real-world decisions.

More data improves explanation consistency: As sample size increased, explanations from different models began to converge and show agreement. This suggests organizations should validate that they have sufficient training data before trusting an AI system's explanations for important decisions like loan approvals or medical diagnoses.

Current explanation methods have limitations: The study used SHAP, a popular explanation technique, but found it still produced variable results at low sample sizes. This indicates that even widely-adopted explanation tools cannot guarantee reliable insights without proper validation conditions.

Validation is essential before deployment: The authors emphasize that "conclusions may be unreliable without validation," suggesting organizations should test whether their AI explanations are stable and consistent before relying on them for high-stakes decisions.