Forecasting AI Progress: A Research Agenda

Forecasting AI progress is critical for safety planning: The paper emphasizes that reducing uncertainty about AI development timelines and capabilities is essential for researchers and policymakers to effectively plan AI safety and governance strategies.

Expert consensus favors a mixed-methods approach: Rather than relying on a single forecasting method, experts identified that combining statistical methods with human judgment produces more reliable predictions about AI progress.

Validating forecasts remains a major unsolved challenge: One of the highest-priority issues identified by experts is determining how to validate AI progress forecasts, suggesting current methods for checking forecast accuracy are incomplete or inadequate.

Forecasts need to drive real-world decisions: Experts highlighted that forecasting AI progress is only valuable if the predictions can be made "action-guiding"—meaning they should directly inform concrete decisions about research priorities and governance policies rather than remaining theoretical exercises.

Performance metrics for AI systems need improvement: The paper identifies that the quality and appropriateness of different metrics used to measure AI capabilities is a critical gap, suggesting current benchmarks may not adequately capture meaningful progress in AI development.