AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation

Clinicians can reliably evaluate AI chatbot safety: Licensed mental health professionals showed strong agreement (77% inter-rater reliability) when rating chatbot responses to users at suicide risk using the VERA-MH rubric, establishing that expert consensus on AI safety is achievable and measurable.

AI can be trained to match expert judgment on mental health safety: An AI-based evaluator achieved 81% agreement with clinician consensus when assessing the same conversations, suggesting that automated safety evaluation tools could potentially scale expert-level assessment across the millions of chatbot interactions occurring daily.

We now have a validated benchmark for AI mental health safety: The VERA-MH evaluation framework, tested against real clinician judgments, provides the first validated automated tool for assessing whether AI chatbots respond safely to users expressing suicidal thoughts—addressing a critical gap in responsible AI deployment for mental health.

Consistency matters for trustworthy AI evaluation: The LLM judge's ratings remained stable across different AI models and repeated evaluations, demonstrating that automated safety assessment can be reliable and reproducible—important for building trust in AI oversight systems.

Realistic testing reveals real safety gaps: By simulating conversations with varying suicide risk levels and disclosure styles, the study identified how different chatbots actually perform with vulnerable users, moving beyond theoretical safety claims to evidence-based evaluation.