Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments

Individual AI assessments of user states are unreliable for real-time decisions: The study found that while AI models (GPT-4o, Gemini 2.0 and 2.5) can measure user states like satisfaction and engagement, individual scores lack the stability needed to make immediate, personalized adaptations in conversational systems—meaning systems shouldn't rely on single AI judgments to adjust responses in real-time.

Aggregated data may mask underlying instability: Metrics that appear unreliable when measured individually can seem stable when combined across multiple interactions. This distinction matters because it determines whether AI assessments are trustworthy for different purposes—real-time adaptation versus later analysis of patterns.

Reliability is not automatic in subjective domains: The research challenges the assumption that AI-inferred measurements of psychological states (like user frustration or trust) are inherently dependable. Organizations deploying these systems should actively test reliability rather than assuming it works by default.

Unstable metrics still have analytical value: Even though individual scores can't be trusted for immediate decisions, the same measurements can help researchers and designers understand what factors influence user experience after the fact, supporting system improvement and design decisions.

Validation frameworks are essential before deployment: The paper demonstrates the need for psychometric testing of AI systems before using them in operational settings, particularly when decisions affect individual users in real-time.