Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing

Expert disagreement on AI safety is systematic, not random: When three psychiatrists evaluated AI responses about mental health using the same training and rubric, their agreement was poor (ICC scores of 0.087–0.295), and one measurement factor actually performed worse than random chance (α = -0.203), indicating structured disagreement rather than simple measurement error.

Disagreement is highest on the most critical safety issues: The study found that experts diverged most on suicide and self-harm responses—precisely the highest-stakes scenarios where consistent evaluation matters most for patient safety, suggesting current human feedback methods may be least reliable where they're needed most.

Expert disagreement reflects genuine philosophical differences, not incompetence: Qualitative interviews revealed that psychiatrists' different evaluations stemmed from coherent but incompatible clinical frameworks (safety-first, engagement-centered, and culturally-informed approaches), meaning the disagreement reflects legitimate professional diversity rather than evaluator error.

Averaging expert opinions may hide rather than solve the problem: When disagreement is aggregated into single labels, the process "effectively erases grounded professional philosophies," creating compromised scores that don't reflect any expert's actual clinical reasoning and may obscure important safety considerations.

Human feedback has fundamental limits for AI safety validation in mental health: The findings suggest that relying on aggregated expert labels alone is insufficient for validating mental health AI systems in high-stakes contexts, indicating a need for alternative or supplementary safety evaluation approaches.