No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes

LLMs appear to "know" their answer accuracy before generating it: Researchers found that by analyzing neural activations immediately after a question is read—but before the model generates any response—they can predict whether the model's answer will be correct with meaningful accuracy, suggesting models have some internal signal about their own reliability.

This internal confidence signal generalizes across different types of knowledge: The predictive accuracy held up not just on the training dataset (trivia questions) but also on diverse out-of-distribution knowledge tasks, indicating this is a genuine internal mechanism rather than a quirk specific to one type of question.

The signal breaks down for mathematical reasoning: Notably, the model's ability to predict its own correctness failed when questions required mathematical reasoning, suggesting LLMs may lack reliable internal confidence signals for certain types of complex tasks—an important limitation for real-world deployment.

This finding could improve AI safety and transparency: Since the researchers could detect when models are likely to be wrong before they answer, this technique could potentially be used to flag unreliable outputs or trigger alternative responses (like "I don't know"), helping make AI systems more trustworthy and honest about their limitations.

The confidence signal is detectable without asking the model directly: The probe outperformed methods that ask models to verbalize their confidence, meaning this internal signal exists whether or not the model explicitly acknowledges uncertainty—relevant for understanding when models might give confident-sounding but incorrect answers.