Assessing reliability of explanations in unbalanced datasets: a use-case on the occurrence of frost events

Explanations from AI models can be unreliable, especially with imbalanced data: The paper highlights that while explainability methods are increasingly required by law and regulation, the explanations they produce aren't always trustworthy—a critical gap when these systems inform high-stakes decisions like predicting frost events that affect agriculture.

Imbalanced datasets are common in high-risk applications but create special challenges: Real-world datasets often have rare events (like frost occurrences or disease diagnoses) that are underrepresented, making both AI predictions and their explanations less dependable for the cases that matter most—the minority class that typically represents the critical outcomes.

Explanation consistency should be tested before relying on AI justifications: The researchers propose evaluating whether explanations remain stable and consistent across similar cases, using metrics to verify that an AI system's reasoning is robust rather than arbitrary—a practical quality check for responsible AI deployment.

Current XAI methods need better evaluation frameworks for real-world conditions: The paper demonstrates that standard explainability techniques may not adequately address the specific challenges posed by imbalanced datasets, suggesting that organizations deploying AI in high-risk domains need tailored assessment approaches beyond standard model validation.