Evaluation Cards for XAI Metrics

Explainability metrics lack standardization: The paper identifies that XAI evaluation metrics are "inconsistently defined, incompletely reported, and rarely validated against common baselines," making it difficult to compare different explainability methods fairly or understand what they actually measure.

Documentation transparency is the core problem: Rather than proposing new metrics, the authors argue that the central issue is insufficient transparency in how existing metrics are reported and validated, which undermines accountability in XAI research.

XAI Evaluation Cards provide a solution: The proposed template requires researchers to explicitly document target properties, metric assumptions, validation evidence, gaming risks, and known failure cases—similar to existing "model cards" used for AI transparency, creating a standardized way to report on explainability methods.

Adoption could enable better research synthesis: By establishing a community norm around this documentation template, researchers would be better able to conduct meta-analyses across studies and compare different XAI approaches on equal footing, reducing fragmentation in the field.

Gaming and failure modes need disclosure: The card specifically requires researchers to identify risks that metrics could be "gamed" or manipulated, and to document known failure cases, helping practitioners understand the real-world limitations of explainability methods before deploying them.