"Do I Trust the AI?" Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Reasoning

Doctors' trust in AI doesn't match actual performance: The study found that physicians' perceptions of what LLMs can do in diagnosis often don't align with how these systems actually perform on standardized tests, suggesting that traditional AI benchmarks alone don't capture what matters most in real clinical settings.

Standard AI tests miss what doctors actually need: Benchmark evaluations focus on predefined tasks, but they fail to reflect the complex clinical reasoning physicians use daily. The research shows that doctors value different aspects of diagnostic capability than what typical AI performance metrics measure.

Miscalibrated trust creates real risks: When physicians either over- or under-estimate an LLM's diagnostic abilities, it can lead to poor clinical decisions. The study emphasizes that building appropriate, well-calibrated trust is essential for safe human-AI collaboration in healthcare.

Physician input should shape AI evaluation: By directly surveying 37 physicians about their perceptions of LLM reasoning capabilities, the research demonstrates that clinical experts' perspectives are crucial for designing better AI systems and evaluation methods—not just computer scientists' benchmarks.

Better collaboration requires understanding perception gaps: The findings suggest that improving trustworthy AI-assisted diagnosis depends on closing the gap between how physicians perceive AI capabilities and what those systems can actually do, which requires ongoing dialogue between AI developers and clinical practitioners.