An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

Worst-case scenarios reveal hidden risks: When AI models are evaluated using worst-case aggregation rather than averages, systemic risk scores drop by 14-37 points, suggesting that average assessments may mask serious potential harms in specific deployment contexts.

Transparency enables informed decision-making: The Systemic Risk Index makes AI safety evidence publicly accessible and traceable through an interactive dashboard, allowing non-experts to understand how specific benchmarks contribute to overall risk ratings rather than relying on opaque claims.

Human-level consistency in AI evaluation: Large language models used as judges achieved agreement with human graders comparable to human-human agreement (κ = 0.78–0.82), suggesting that automated evaluation tools can reliably assess AI harms when properly designed.

Context matters for risk assessment: The pipeline evaluates models under simulated deployment contexts and uses "harm-preserving perturbations" to test robustness, demonstrating that AI safety cannot be assessed in isolation but must account for how systems will actually be used.

Flexible evaluation frameworks support responsible governance: The dashboard's ability to adjust aggregation methods, weighting schemes, and evidence sources empowers stakeholders—regulators, developers, and the public—to evaluate AI systems according to different priorities and risk tolerances under frameworks like the EU AI Act.