How much informative is your XAI? A decision-making assessment task to objectively measure the goodness of explanations

We need better ways to measure if AI explanations actually work: The paper identifies a critical gap in XAI research—most systems are evaluated indirectly through performance metrics rather than measuring whether explanations genuinely help users understand AI decisions. This means we may be deploying "explanations" that don't actually explain.

User-centered explanations show promise but lack objective measurement: While recent research suggests that personalized, user-focused explanation approaches improve human-AI interaction, the paper argues these improvements haven't been rigorously quantified. The authors propose a new assessment task to objectively measure how much useful information explanations actually convey to users.

Information power is a measurable standard for explanation quality: The paper introduces "information power"—the amount of information a system provides during user interaction—as a concrete, quantifiable metric for assessing XAI goodness. This offers a more direct way to evaluate whether one explanation technique is genuinely better than another.

Human-robot decision-making tasks can reveal explanation effectiveness: By testing XAI techniques in real decision-making scenarios (rather than abstract settings), researchers can better understand whether user-centered approaches truly outperform traditional explanation methods in practical contexts where it matters.

Responsible AI requires moving beyond assumptions about explanations: The paper highlights that the XAI field has assumed certain approaches work better without rigorous proof, suggesting that responsible AI deployment demands objective evidence that explanations actually help users make better decisions.