RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

AI agents are approaching human-level performance on complex ML research tasks: When given a 2-hour time budget, the best AI agents scored 4x higher than human experts on RE-Bench's research engineering challenges, suggesting frontier AI models have developed substantial capabilities in autonomous R&D work.

Humans retain advantages in long-horizon problem-solving: Despite AI agents' initial speed advantage, human experts showed better returns on increased time investment, narrowly exceeding top AI agent scores at 8 hours and achieving 2x the score at 32 hours, indicating humans excel at sustained, iterative research over longer periods.

AI agents demonstrate specialized technical expertise: The paper documents instances where AI agents outperformed all human experts—such as writing a faster custom Triton kernel than any of the 61 human experts tested—showing frontier models have acquired deep, specialized knowledge in specific ML domains.

Speed and cost efficiency create new automation risks: AI agents can generate and test solutions over 10 times faster than humans at much lower cost, which could accelerate AI R&D timelines in ways that outpace human oversight capabilities, even if humans eventually produce better final results.

Current benchmarks reveal a critical capability gap requiring policy attention: The existence of realistic R&D automation benchmarks like RE-Bench is itself significant, as frontier AI safety policies identify AI-driven R&D automation as an important capability to anticipate and monitor.