Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety

Real-world conversations reveal gaps that simulations miss: Testing on 20,000 actual deployment conversations uncovered safety issues that benchmark tests alone would not have caught. For example, the audit found 3 conversations about non-suicidal self-injury (NSSI) that lacked crisis intervention resources, demonstrating that small simulation-based tests may not capture the full range of scenarios that occur in practice.

Purpose-built mental health AI significantly outperforms general-purpose models: The specialized mental health system produced harmful content rates of 6.2% on suicide/self-harm topics, compared to 18.0-52.0% across frontier general-purpose models like GPT-5 variants, DeepSeek V3, and Google Gemini 3 Flash—suggesting that AI systems designed specifically for mental health contexts perform substantially better on safety metrics.

The system successfully identified and responded to crisis situations at scale: When clinicians reviewed 600 randomly sampled conversations, the AI correctly identified all 6 confirmed suicide-risk cases and provided crisis resources in every instance, achieving 100% sensitivity with no false negatives—indicating reliable performance on the most critical safety outcomes.

Ecological auditing should complement traditional benchmarking: The paper argues that real-world deployment audits are essential for responsible AI evaluation in mental health, as they capture linguistic diversity, contextual complexity, and actual user behavior that controlled benchmarks cannot replicate, suggesting this approach should become standard practice.