Do LLMs estimate uncertainty well in instruction-following?

LLMs struggle to accurately assess their own confidence when following instructions, particularly when making subtle errors—meaning a model might confidently provide an incorrect response without signaling uncertainty to users, a critical problem for high-stakes applications like healthcare or legal advice.

Current methods for measuring model uncertainty are inadequate, with the paper finding that even approaches using internal model states (the model's internal computations) fail to reliably detect when the model has misunderstood or incompletely followed user instructions.

Existing instruction-following benchmarks mix multiple sources of error together, making it difficult to understand which specific problems cause models to fail—the researchers created a more controlled testing environment to isolate instruction-following uncertainty from other factors.

The gap between LLM capabilities and trustworthiness remains significant for real-world deployment, as models cannot reliably tell users when they're uncertain about whether they've correctly understood what was asked, undermining their reliability as personal AI agents in important domains.

Better uncertainty estimation methods are essential before widespread deployment, since the current inability to detect when models are likely wrong means users cannot know when to seek alternative information or human verification.