A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI

AI can automate tedious software quality checks: The paper demonstrates that Large Language Models can automatically classify Non-Functional Requirements (NFRs)—specifications about software performance, usability, and reliability—reducing the manual work that is "time-consuming and prone to errors."

Different AI models perform differently on the same task: The study compared six different language models and found significant variation in their accuracy. Gemma-2 achieved the best results with 87% precision and 89% recall, while Phi-3 came close behind, showing that model selection matters for real-world applications.

Explainability is measurable and important: The researchers used "lime scores" to evaluate how well each model could explain its classification decisions (Phi-3 scored 79 out of 80). This demonstrates that responsible AI implementation should include tools to understand why an AI system made a particular decision, not just what decision it made.

Quality datasets are foundational for responsible AI: The authors built their benchmark by collecting requirements from real software projects and open-source documentation rather than using synthetic data. This emphasis on comprehensive, real-world datasets reflects best practices for developing AI systems that work reliably in practice.

Context and domain knowledge improve AI performance: The paper notes that "improving the contextual foundation" and providing models with better technical background information "enhanced the model's comprehension," suggesting that responsible AI deployment requires careful preparation of training materials and domain expertise.