Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook

Current LLM evaluation metrics miss the human element: The paper argues that standard metrics used to evaluate large language models (LLMs) like accuracy and performance scores don't account for how actual humans interact with these systems or whether they meet real user needs—a gap that the explainable AI (XAI) field has already recognized and addressed over the past decade.

Human cognitive biases should shape how we test AI systems: Rather than treating humans as neutral evaluators, the research emphasizes that people have inherent cognitive biases and quirks that affect how they understand and trust AI explanations. These human factors should be central to evaluation design, not treated as afterthoughts.

Three critical areas need human-centered testing: The paper identifies three key focus areas for evaluating LLMs responsibly: (1) mental models—whether users actually understand how the system works, (2) use case utility—whether the system solves real problems for its intended users, and (3) cognitive engagement—how users interact with and process the system's outputs.

The XAI field offers a proven roadmap: The explainable AI community has spent over a decade developing human-centered evaluation methods. The paper warns that LLM research risks repeating the same mistakes by ignoring these lessons, suggesting the field should adopt similar human-focused approaches rather than starting from scratch.

Deployed AI requires evaluation beyond lab metrics: Since LLMs impact real people in real-world settings, evaluation must combine both quantitative data and qualitative human feedback to ensure systems actually serve users' needs responsibly.