Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

LLMs show inconsistent personality stability over time: The study found "varying levels of inter-rater agreement" across seven different LLMs when tested on personality instruments at two different time points, meaning the same model may give different personality-related responses when tested again shortly after, raising concerns about reliability.

Performance varies significantly between models: Some models like Llama3 and GPT-4o demonstrated higher consistency in their responses, while GPT-4 and Gemini showed lower agreement, suggesting that personality stability is not a universal feature across LLMs and depends on the specific model being used.

LLMs present idealized, socially desirable personalities: When personality tests showed fair agreement, LLMs consistently displayed "socially desirable" profiles with higher agreeableness and conscientiousness while showing lower Machiavellianism—essentially presenting themselves as more ethical and prosocial than typical humans, which may not reflect their actual capabilities or limitations.

Measurement challenges depend on the assessment tool: The study found that "agreement depended on used instruments as well as on domain or trait," meaning some personality dimensions are easier to measure consistently in LLMs than others, complicating efforts to reliably assess AI systems.

Personality consistency matters for AI safety: The researchers emphasize that "exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns," suggesting that unpredictable personality-like behaviors could pose risks as these systems become more integrated into society.