Trusting the Explainers: Teacher Validation of Explainable Artificial Intelligence for Course Design

Explainable AI methods don't always agree on what matters: The study found that LIME and SHAP, two popular explanation techniques, "significantly disagree with each other about what is important" when analyzing student success prediction models, raising questions about which explanations educators should actually trust.

Expert educators themselves lack consensus on trustworthiness: When 26 university instructors were interviewed about the explanations, they did not agree on which explanation method was most trustworthy, suggesting that transparency alone doesn't automatically build confidence in AI systems for educational decisions.

Black-box AI adoption in education is hindered by trust gaps: The research identifies lack of trust and transparency as likely reasons why deep learning models for learning analytics "are still not widely adopted in real-world settings," even though they've become increasingly popular in research.

Explainable AI has potential for actionable course design: Despite disagreements about trustworthiness, educators indicated they could "transform these insights into actionable course design decisions," suggesting that explainability methods could support real educational improvements if trust issues are resolved.

One-size-fits-all explanations may not work for educators: The study's focus on controlled course comparisons reveals that what explains student success may vary across different educational contexts, implying that explainable AI systems need to account for course-specific factors rather than providing universal explanations.