Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors

Counterfactual explanations for AI systems lack reliable evaluation methods: The paper identifies a critical gap in responsible AI—while counterfactual editors (tools that generate explanations by showing how input changes affect predictions) are increasingly used, there is "no universal ground truth" to evaluate whether they're working correctly, making it difficult to trust their explanations.

Different explanation tools behave inconsistently: The researchers found that various counterfactual explainers "can vary significantly" in their outputs, meaning the same AI model might receive different explanations depending on which tool is used—a concerning finding for anyone relying on these explanations to understand AI decisions.

A new "back-translation" testing method reveals hidden patterns: By feeding counterfactual explanations back into the explainer iteratively, the authors discovered "valuable insights into the behaviour of both the predictor and the explainer models" that would otherwise remain hidden, suggesting this approach could become a standard way to audit explanation tools.

Consistency should be a key metric for trustworthy AI explanations: The paper proposes measuring whether counterfactual editors produce consistent, reliable explanations across different scenarios—establishing consistency as an important criterion for determining whether an explanation tool can be trusted in real-world applications.

This work addresses a practical barrier to responsible AI deployment: By developing better evaluation methods for explanation tools, this research helps organizations verify that their AI systems can be meaningfully explained to stakeholders and users, a prerequisite for accountable AI systems.