Evaluating Model Capabilities After Benchmark Saturation

Benchmarks become less useful as AI models improve: When multiple AI systems achieve near-perfect scores on standard tests, these benchmarks can no longer distinguish between different model capabilities—a critical problem for safety evaluation and policy decisions.

Model transcripts contain hidden capability signals: Rather than discarding benchmark data once scores saturate, researchers found that detailed transcripts of how models solved problems retain valuable information about their true capabilities, even when final scores no longer differentiate between systems.

Comparing model responses works better than scoring them individually: Among the methods tested, having evaluators compare pairs of model transcripts was 5.3 times more effective at predicting real capabilities than relying solely on saturated benchmark scores, suggesting comparative evaluation is more informative than absolute scoring.

Specific types of reasoning matter most: The capability information in transcripts is concentrated in particular areas—especially a model's reasoning process and how it uses tools—rather than being evenly distributed across all parts of the response.

New evaluation methods need careful validation: While transcript analysis shows promise for extending benchmark utility, the researchers note that predictive power doesn't always transfer reliably across different contexts, suggesting these approaches require ongoing testing before deployment in real safety assessments.