Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs

Chain-of-Thought prompting improves accuracy but creates overconfidence: While this popular technique enhanced medical question performance, it simultaneously made models more confident than their actual accuracy warranted, potentially leading clinicians to trust incorrect answers in real medical decisions.

Emotional prompting poses serious risks in medical contexts: The study found that prompting models to respond with emotional language further inflated confidence levels beyond Chain-of-Thought alone, creating a compounding problem where models become increasingly unreliable precisely when they seem most trustworthy.

Smaller, open-source models underperform significantly: Llama-3.1-8b consistently underperformed across all evaluation metrics compared to proprietary models like GPT-4o and o3-mini, suggesting that cost-effective alternatives may not be suitable for high-stakes medical applications without substantial additional safeguards.

Proprietary models still lack reliable confidence calibration: Even the best-performing models (GPT-4o, o3-mini) showed misalignment between their stated confidence and actual accuracy, meaning users cannot safely rely on a model's confidence score to judge whether its medical advice is trustworthy.

Prompt engineering alone cannot solve the confidence problem: The research tested 156 different configurations but found no single approach that reliably produced both accurate answers and well-calibrated confidence, indicating that additional technical solutions beyond prompt design are needed before deploying medical LLMs safely.