What if automating AI R&D triggers an intelligence explosion?

AI systems are already automating their own development: The paper notes that AI systems now write most code inside major AI companies, and are on track to automate most or all AI R&D work within a few years—a significant acceleration from just one year prior.

An "intelligence explosion" could compress years of progress into months: If AI automation of R&D triggers a feedback loop where AI systems rapidly improve themselves, technological advancement could accelerate dramatically beyond current trajectories, potentially outpacing society's ability to manage the transition.

The risks include loss of human control and erosion of institutional checks: Beyond capability misalignment, an intelligence explosion could undermine governmental oversight, corporate accountability, and the balance of power between institutions—creating governance challenges that existing regulatory frameworks may not address.

Benefits could arrive faster, but so could dangers: While accelerated AI progress could bring societal benefits sooner, the compressed timeline means less opportunity for safety testing, policy development, and public adaptation, amplifying potential harms.

Policymakers need immediate action on visibility and steering mechanisms: The paper calls for urgent policy responses including greater transparency into AI R&D automation, development of methods to constrain an intelligence explosion, and societal preparation for rapid change—emphasizing that uncertainty about these risks is not a reason to delay attention.