Quantifying True Robustness: Synonymity-Weighted Similarity for Trustworthy XAI Evaluation

Standard evaluation metrics for XAI robustness are inadequate: The paper demonstrates that commonly used information retrieval metrics fail to account for semantic meaning when assessing adversarial attacks on AI explanations, potentially giving a false impression of how vulnerable these systems actually are.

Synonymity matters in measuring attack success: By incorporating semantic similarity into evaluation metrics, the researchers show that replacing a word with a synonym should not be weighted the same as replacing it with an unrelated word—a distinction that standard metrics miss and that significantly affects vulnerability assessments.

Current robustness assessments may overstate attack success: The paper's key finding is that existing evaluation methods tend to overestimate how damaging adversarial attacks are to XAI systems, which could lead organizations to incorrectly believe their AI explanations are less trustworthy than they actually are.

Better evaluation tools are essential for trustworthy AI: By proposing synonymity-weighted similarity measures, the authors provide a more accurate method for assessing whether AI explanations remain reliable when faced with adversarial manipulation—a critical requirement for responsible AI deployment.

This work addresses a gap in XAI accountability: The paper highlights that trustworthiness evaluation of explainable AI systems has lacked appropriate measurement tools, and improving these assessments is necessary for organizations to genuinely understand and communicate the reliability of their AI systems to stakeholders.