Explainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work?

AI models for code focus on specific code elements: The study found that CodeBERT and GraphCodeBERT place highest attention on particular types of code tokens (like identifiers and keywords) when performing tasks like code documentation and translation, revealing that these models learn to prioritize certain structural elements over others.

Models struggle predictably on certain problems: The research identified common patterns in failure cases—situations where the models perform poorly even on relatively simple coding problems—suggesting these failures are not random but follow identifiable patterns that could be addressed through targeted improvements.

Explainability methods reveal hidden model behavior: By using attention mechanisms as an explainability tool, the researchers were able to "look inside" these AI models to understand what they actually learn, demonstrating that we can make AI systems more transparent rather than treating them as "black boxes."

One-size-fits-all approaches are insufficient: The paper emphasizes that simply deploying the latest neural network model without understanding why it works or fails on specific tasks is inadequate—responsible development requires deeper analysis of model behavior across different scenarios.

Practical recommendations emerge from analysis: The study provides actionable recommendations to address observed challenges in model performance, suggesting that explainability research can directly inform improvements to AI systems used in software engineering.