If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques

User testing is critically lacking: Only 21% of the 100 counterfactual explanation methods surveyed have been tested with actual users, meaning most techniques lack evidence that they actually help people understand AI decisions in practice.

Evaluation standards are inconsistent: The paper identifies five key deficits in how these methods are evaluated, indicating that researchers are using different metrics and approaches, making it impossible to fairly compare which counterfactual explanation techniques work best.

Psychological validation is missing: Beyond the low user testing rate, most counterfactual methods have not been evaluated for whether they genuinely improve human understanding or decision-making—a critical gap since these explanations are meant to be understandable to non-technical users.

Progress in the field is being blocked: The authors argue that the lack of standardized benchmarks and rigorous evaluation is effectively preventing scientific advancement in counterfactual XAI, suggesting the field needs to establish common evaluation frameworks before developing new methods.

Standardized benchmarks are needed: The paper proposes a roadmap for resolving these issues through standardized benchmark evaluations, implying that the field would benefit from agreed-upon tests that all counterfactual explanation methods should undergo before being considered reliable for real-world use.