On the Evaluation Consistency of Attribution-based Explanations

Attribution explanation methods produce inconsistent rankings depending on evaluation settings: The research found that when the same explanation methods are tested under different configurations (different models, datasets, or evaluation protocols), they don't consistently rank the same way. This means a method that appears "best" in one scenario may perform poorly in another, raising questions about which explanations we can actually trust.

Evaluation consistency matters more within a single training process than across different models: While attribution methods show inconsistent performance across different models and datasets, they do maintain relatively stable performance rankings when evaluated at different points during a single model's training. This suggests that how we benchmark these methods is critical to drawing reliable conclusions.

Current standardized evaluation approaches don't solve the consistency problem: Previous attempts to create consistent evaluation frameworks for attribution methods don't perform better than simpler baseline approaches when tested across more diverse models and datasets, indicating that the inconsistency problem is more fundamental than existing solutions address.

The field needs more rigorous and comprehensive testing: The research emphasizes that attribution-based explanations—which are becoming the main approach to making AI systems explainable—require evaluation across a much broader range of real-world models and datasets before we can confidently claim any method is reliable for explaining AI decisions.

We may need to reconsider why some attribution methods appear to work: The findings suggest that the assumptions underlying why certain explanation methods seem successful may need to be re-examined, as their apparent success might be an artifact of limited or inconsistent testing rather than genuine effectiveness.