Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

Rigorous evaluation standards are urgently needed: The paper identifies that AI evaluation research currently lacks common standards and shared vocabulary, making it difficult to produce cumulative, comparable evidence that policymakers can rely on—a critical gap for responsible AI governance.

Five core principles should guide AI studies: The framework synthesizes five foundational principles drawn from established RCT traditions in clinical trials, economics, psychology, and software engineering, focusing on transparency, repeatability, and verification to ensure AI evaluation research is trustworthy and reproducible.

33 actionable guidelines provide practical implementation: Rather than abstract principles alone, the paper translates these into 33 specific, implementable guidelines complete with rationales and evidence bases, designed to help researchers plan studies, assess existing work, and establish field-wide standards.

AI evaluation can learn from proven methodologies: By adapting rigorous RCT practices from disciplines with long-standing traditions (clinical medicine, economics, psychology), the framework demonstrates that responsible AI evaluation doesn't require starting from scratch—established best practices can be adapted and standardized for AI contexts.

Standardization serves multiple purposes simultaneously: The framework functions as a design tool for planning new studies, an evaluation rubric for assessing published research, and a blueprint for industry-wide norm-setting, positioning standardized evaluation as foundational to building trustworthy AI systems.