Toward Explaining Large Language Models in Software Engineering Tasks

LLMs in software engineering lack transparency: The paper identifies that while Large Language Models have advanced automation of critical SE tasks like code generation and summarization, their "black-box nature remains a major barrier to their adoption in high-stakes and safety-critical domains," highlighting a key gap between capability and trustworthiness.

Existing explainability methods don't match how engineers think: Current approaches to explaining AI systems fail to provide "domain-specific explanations aligned with how practitioners reason about SE artifacts," meaning generic AI explanations aren't sufficient for software engineering contexts where domain expertise matters.

FeatureSHAP provides automated, practical explanations: The paper introduces FeatureSHAP, a framework that automatically explains which input features drive LLM outputs for code tasks, and importantly, it works with both open-source and proprietary models—addressing real-world deployment scenarios.

Practitioner feedback validates the approach: A survey of 37 software engineers found that FeatureSHAP "helps practitioners better interpret model outputs and make more informed decisions," demonstrating that improved explainability translates to tangible benefits for human oversight and decision-making.

Better explanations enable safer AI adoption: By reducing reliance on irrelevant features and improving explanation fidelity compared to baseline methods, FeatureSHAP represents progress toward making LLMs trustworthy enough for safety-critical software engineering applications where accountability is essential.