Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection

One tool doesn't fit all contexts: The study evaluated 14 different sentiment analysis tools across 10 datasets and found that "tool effectiveness remains context-dependent," meaning software teams cannot simply pick one tool and expect it to work equally well across different communication platforms like Slack, GitHub, or Stack Overflow.

Platform differences matter significantly: Developer communication varies substantially between platforms in both linguistic style and statistical properties. This means a tool trained on one platform's data may perform poorly when analyzing sentiment from another platform, which is critical for teams using multiple communication channels.

Modern AI models show promise but need verification: Transformer-based models like SetFit and RoBERTa "consistently achieve strong results," suggesting newer AI approaches are more reliable than older methods—but the authors emphasize that even these strong performers still require context-specific evaluation before deployment.

Dataset characteristics can guide tool selection: Rather than guessing which tool to use, the researchers propose a practical questionnaire approach that matches a dataset's linguistic and statistical features to appropriate sentiment analysis tools, helping practitioners make more informed, trustworthy choices.

Sentiment analysis in software engineering requires ongoing oversight: The paper emphasizes "the need for ongoing evaluation as communication contexts evolve," highlighting that responsible AI implementation means continuously monitoring tool performance rather than assuming a one-time evaluation is sufficient.