LLM-based Models for Detecting Emerging Topics in Service Feedback

Human oversight is critical for AI reliability: The study found that incorporating human-in-the-loop review significantly reduced LLM "fabrication" (false or misleading outputs) and improved alignment with expert judgments, suggesting that AI systems analyzing public sector feedback should never operate without human expert validation.

AI can help identify service inequities at scale: By analyzing large volumes of multilingual customer feedback, the LLM-based approach was able to detect emerging service quality issues and potential disparities across diverse populations—tasks that would be impractical through manual review alone, making this particularly valuable for ensuring fair public service delivery.

Traditional static approaches miss important patterns: The paper highlights that relying on manual review or pre-defined expert indicators limits organizations' ability to capture complex, evolving issues in feedback. AI-assisted analysis can adapt to identify new problems as they emerge rather than only detecting predetermined categories.

Technical efficiency matters for public sector adoption: The framework uses "fine-tuned, quantized LLMs" (smaller, optimized models) rather than massive general-purpose systems, making the approach computationally efficient and more practical for government organizations with limited IT resources.

Context and expertise must shape AI design: The methodology was specifically tailored to tax administration feedback and validated by experienced tax officers, demonstrating that responsible AI implementation requires domain expertise and customization rather than applying generic off-the-shelf solutions.