CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs

Detailed behavioral data is essential for responsible AI research: CodeWatcher enables researchers to capture fine-grained interaction events (insertions, deletions, copy-paste actions, focus shifts) that reveal how and when developers actually use AI code generation tools, moving beyond surface-level metrics to understand real-world coding practices.

Privacy-conscious monitoring is possible without disrupting workflows: The tool is designed as "lightweight" and "unobtrusive," capturing meaningful data while preserving developer experience—an important consideration for responsible AI deployment, as it demonstrates that oversight and research can occur without intrusive surveillance that might undermine trust.

Infrastructure for human-centered AI evaluation: By enabling "post-hoc reconstruction of coding sessions" and "rich behavioral analyses," CodeWatcher provides the technical foundation needed to evaluate code generation tools from a human perspective, supporting research on developer productivity and responsible AI practices rather than relying solely on automated metrics.

Open science and reproducibility: The authors make their tool, diagrams, and demo publicly available through an open repository, facilitating broader research community access to standardized methods for studying AI-developer interactions—critical for building shared understanding of responsible AI implementation across organizations.