Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces

Correct answers don't guarantee valid reasoning: When tested on harder math problems, 31.6% of correct answers were accompanied by invalid reasoning traces, meaning models can arrive at the right answer through flawed or nonsensical intermediate steps—a critical finding for anyone relying on chain-of-thought explanations to understand how AI systems work.

AI reasoning traces may not reflect actual computation: Over half of the invalid traces passed basic syntax and arithmetic checks but failed when examined for logical dependencies, suggesting that chain-of-thought explanations can appear plausible on the surface while masking fundamental problems in how models actually process information.

Training data directly shapes explanation patterns, not reasoning ability: When researchers deliberately corrupted training traces (shuffling words, swapping steps), models maintained high accuracy while producing nonsensical explanations, indicating that models learn to mimic explanation formats rather than developing genuine reasoning processes.

Current verification methods are insufficient: Syntactic and arithmetic checks alone cannot catch invalid reasoning—semantic dependency verification is necessary, but this level of scrutiny is rarely applied in practice, meaning flawed traces often go undetected in real-world AI auditing.

Chain-of-thought traces should not be trusted for AI auditing: The decoupling between answer correctness and trace validity undermines the use of reasoning traces for debugging, agent oversight, and claims about AI reasoning capabilities, suggesting current practices for validating AI decision-making are inadequate.