Scaling Vision Models Does Not Consistently Improve Localisation-Based Explanation Quality

Bigger AI models don't automatically produce better explanations: The study tested 11 computer vision models of increasing size and found that "increasing architectural depth and parameter count does not improve explanation quality in most statistical comparisons," with smaller models often performing equally well or better at explaining their decisions.

High accuracy doesn't guarantee trustworthy reasoning: The researchers discovered "scenarios in which models achieve strong predictive performance while localisation precision is near zero," meaning a model can make correct predictions while basing those decisions on the wrong visual features—a critical finding for assessing whether AI systems are truly reliable.

Current explanation methods have limitations: Using five different explanation techniques across three datasets, the study found inconsistent results in how well these methods could identify which parts of images the models actually used for their decisions, suggesting that popular explainability tools may not be as dependable as assumed.

Pretraining doesn't reliably improve explanation quality: While pretraining (training on large datasets first) typically improves prediction accuracy, it "does not consistently increase localisation scores," indicating that the standard practice of using pretrained models may not enhance our ability to understand why models make decisions.

Performance metrics alone are insufficient for responsible AI: The findings suggest that relying only on accuracy scores to evaluate AI systems is inadequate—organizations should also directly measure whether models are using the right reasoning, not just achieving correct outputs.