An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mapping

Evaluation must extend beyond AI models themselves. The paper emphasizes that AI safety evaluation cannot focus solely on the underlying models—it must consider the entire system, including how models are deployed, what tools they can access, and the broader context in which they operate.

Different communities speak different languages about AI safety. AI researchers, software engineers, and policymakers use divergent terminology and practices when evaluating AI systems, creating communication barriers that hinder comprehensive safety assessment. The framework proposes harmonized terminology to bridge these gaps.

A shared evaluation framework is essential for accountability. The paper maps AI system lifecycles to specific stakeholders and required evaluations, establishing clear responsibility points throughout the supply chain—from development through deployment—to ensure no safety gaps fall through the cracks.

Taxonomy of evaluation elements provides a common foundation. By identifying and categorizing essential elements needed for AI system evaluation, the framework gives organizations and regulators a structured approach to ensure they're addressing all relevant safety considerations systematically.

Collaboration across disciplines is urgent and necessary. The paper underscores that advancing AI safety requires coordinated effort among AI developers, software engineers, and governance experts—no single community can adequately evaluate AI systems in isolation.