On the Reliability and Explainability of Language Models for Program Generation

Data duplication inflates performance claims: The study found that state-of-the-art language models for code generation (like CodeT5 and CodeGPT) appear more reliable than they actually are due to severe data duplication in benchmark datasets, leading to "over-optimistic results" that don't reflect real-world performance.

Models struggle with input variations: While language models can recognize code grammar and structure, the explainability analysis revealed they have "limited robustness to changes in input sequences"—meaning small modifications to code can cause the models to fail, raising concerns about their reliability for practical deployment.

Current evaluation methods are inadequate: The paper emphasizes that "more rigorous evaluation approaches and benchmarks are critical," suggesting that existing testing methods don't sufficiently assess whether these automated code generation tools are trustworthy enough for real-world use.

Explainability reveals model limitations: By analyzing which parts of code the models focus on, researchers discovered that while models understand structural patterns, they lack the robustness needed for consistent, reliable program transformation across different scenarios.

Trustworthiness remains uncertain: The core finding questions whether current language model-based code generation techniques are "sufficiently trustworthy for automated program generation," indicating that organizations should be cautious about deploying these tools in critical applications without further validation.