EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks

Popular explainability tools like LIME and PyExplainer produce inconsistent and unreliable explanations for machine learning models used in software engineering tasks, meaning practitioners cannot always trust the reasons these tools give for why a model made a particular prediction.

The paper introduces EvaluateXAI, a new framework designed to automatically test whether explainability techniques are actually generating trustworthy explanations, addressing a gap where these tools were previously evaluated only through manual inspection.

Inconsistencies were found across multiple real-world software engineering applications, including defect prediction, code clone detection, and code review analysis, suggesting the problem is widespread rather than limited to specific use cases.

None of the six evaluation metrics tested achieved perfect reliability scores, indicating that current rule-based explainability techniques have fundamental limitations in consistently explaining ML model decisions—a critical concern for practitioners who need to trust AI recommendations in software development.

The research highlights a critical gap between ML model adoption and real-world deployment: even when ML models work well, organizations hesitate to use them because they cannot understand or verify the model's reasoning, and current explainability tools do not fully solve this trust problem.