Learning To Guide Human Decision Makers With Vision-Language Models

Human oversight remains critical in high-stakes AI: The paper identifies a key problem with current AI-assisted decision-making systems—experts may over-rely on AI recommendations due to anchoring bias, undermining the human oversight that regulators increasingly require for trustworthy AI deployment.

Guidance, not automation, is the better approach: Rather than having AI systems make decisions autonomously while humans handle edge cases, the "Learning to Guide" framework keeps humans fully responsible for final decisions while AI provides interpretable textual guidance to support their judgment.

Vision-language models can be adapted for explainability: The SLOG approach demonstrates that existing vision-language models can be fine-tuned with minimal human feedback to generate task-specific, interpretable explanations—making AI assistance more transparent and trustworthy in domains like medical diagnosis.

The hardest decisions shouldn't be left unassisted: Current systems often leave experts entirely on their own for the most difficult cases where the model abstains. The proposed framework ensures humans receive AI guidance precisely when they need it most, rather than only on routine decisions.

Real-world medical testing shows promise: The empirical evaluation on both synthetic and real-world medical diagnosis tasks suggests this human-centered guidance approach is practical and effective for high-stakes professional decision-making.