Faster but Not Wiser: GitHub Copilot Decouples Programming Performance from Code Comprehension in Brownfield Tasks

Speed gains don't equal understanding: While GitHub Copilot significantly improved how quickly students completed programming tasks, it did not improve their actual comprehension of the existing code (p=0.59), suggesting that faster task completion is a misleading indicator of learning.

AI assistance may discourage deep code inspection: Students who achieved better comprehension engaged in more "verification loops"—repeatedly inspecting and revising code—performing write-then-view transitions 4.7 times more often than those with lower comprehension (p=0.001), indicating that AI tools may reduce the careful review behaviors necessary for understanding.

Different comprehension challenges require different approaches: While Copilot helped students identify what and where to modify code, it appeared to hinder their ability to explain how existing code worked and predict the effects of changes (ρ=-0.57), suggesting AI assistance has uneven impacts across different types of programming knowledge.

Task completion is not a reliable measure of learning: The disconnect between performance improvements and comprehension gains reveals that metrics commonly used to evaluate AI tools in education—like task speed and correctness—may mask important gaps in student understanding that could cause problems later.

How students interact with AI matters more than whether they use it: The study suggests that the behavioral patterns students develop when using AI tools—particularly whether they actively verify and inspect generated code—may be more important for learning outcomes than simply having access to the tool.