Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications

LLMs can exhibit consistent personality patterns, but only under specific conditions: The survey finds that while some large language models show reproducible psychological traits when prompted in particular ways, this consistency is not guaranteed across different tasks and settings, suggesting that LLM "personality" may be more fragile and context-dependent than human personality.

Current psychological testing tools weren't designed for AI and may not work well: The paper identifies a fundamental mismatch between traditional psychological assessment methods (developed for humans) and LLM capabilities, highlighting that researchers need to develop new evaluation frameworks specifically designed for AI systems rather than simply adapting human psychology tests.

There's no standardized way to measure LLM psychology yet: The survey reveals significant inconsistencies in how different researchers evaluate psychological traits in LLMs, including variations in evaluation metrics and practices, which makes it difficult to compare findings across studies and establish reliable benchmarks for trustworthy AI.

Assessing LLM psychology matters for real-world safety: As these models take on increasingly human-centered roles (like counseling, education, or customer service), understanding their psychological characteristics becomes essential for predicting their social impact and ensuring they can be trusted in sensitive applications.

We need more robust and interpretable assessment methods going forward: The paper calls for developing psychological evaluation frameworks that are more generalizable, transparent, and specifically tailored to how LLMs actually work, rather than relying on adapted human psychology measures that may not capture what's really happening inside these systems.