A Detailed Study of Interpretability of Deep Neural Network based Top Taggers

Different interpretability methods can give conflicting results: The paper finds that various explainable AI (XAI) metrics identify different features as important for the same task, highlighting a critical pitfall—researchers cannot simply pick one interpretation method and assume it reveals the true decision-making process of a neural network.

Feature correlations complicate interpretability: When input features are correlated with each other, it becomes harder to accurately determine which features actually matter for the model's decisions, suggesting that interpretability analysis requires careful consideration of data structure, not just model analysis.

Understanding hidden layers enables model simplification: By visualizing how information flows through neural networks using Neural Activation Pattern diagrams, researchers can redesign and optimize these models to be simpler and more efficient without sacrificing performance—making them potentially easier to audit and deploy responsibly.

Interpretability requires multiple complementary approaches: The paper demonstrates that no single XAI method is sufficient; instead, combining different interpretation techniques and validating findings against physical knowledge produces more reliable and meaningful explanations of how deep learning models actually work.

Latent representations encode meaningful information: The study shows that the internal "learned representations" in neural networks correlate with real-world physical quantities, suggesting these models may be learning genuine patterns rather than spurious correlations—an important validation for trustworthiness.