Investigating the Impact of Model Instability on Explanations and Uncertainty

Explanations can be unreliable even when AI models perform well: Small, hard-to-detect changes to input text can dramatically distort the explanations AI systems provide for their decisions, yet these flawed explanations may not be caught during standard pre-deployment testing since evaluations typically happen before the model is in use.

Uncertainty measures alone cannot guarantee trustworthy explanations: Contrary to intuition, when an AI model reports high uncertainty about its prediction, this does not necessarily mean its explanation is unreliable. The researchers found only moderate correlation between uncertainty levels and explanation quality, suggesting that confidence scores should not be the sole basis for trusting an explanation.

Training models with noisy data may improve explanation reliability: Models trained on deliberately corrupted or noisy text became better at identifying which words were actually important for their decisions when facing uncertain situations, suggesting that exposure to imperfect data during training could strengthen explanation robustness.

Different explanation methods have varying stability: The study found that Integrated Gradients, one common explanation technique, showed greater resistance to input perturbations compared to other methods tested, though performance still varied depending on the specific model being explained.

Overconfident uncertainty estimates signal potential problems: When AI systems are overconfident in their uncertainty measurements, the stability of their explanations under small input changes can serve as a warning sign that the model may have underlying stability issues worth investigating before deployment.