AccountabilityFairnessSafetycs.CL
COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act
Philipp Guldimann, Alexander Spiridonov, Robin Staab, Nikola Jovanović, Mark Vero, Velko Vechev, Anna-Maria Gueorguieva, Mislav Balunović, Nikola Konstantinov, Pavol Bielik, Petar Tsankov, Martin Vechev
October 10, 2024
arXiv.org
35 citations
3 influential
Key Takeaways
-
The EU AI Act's regulatory requirements are too vague for practical compliance assessment, prompting the researchers to create the first technical interpretation that translates broad legal obligations into measurable technical standards specifically for large language models.
-
Testing 12 prominent LLMs against the COMPL-AI framework revealed significant shortcomings across multiple critical areas: robustness, safety, diversity, and fairness—suggesting current models may not adequately meet the EU's regulatory expectations.
-
The paper demonstrates that translating legal requirements into concrete technical benchmarks is both possible and necessary, but also challenging, indicating that regulators and AI developers need better tools and shared standards to operationalize compliance.
-
Existing LLM benchmarks are insufficient for assessing regulatory compliance, highlighting a gap between current evaluation practices and what's needed to verify that AI systems meet responsible AI standards.
-
This framework could provide actionable guidance for AI model providers on how to develop compliant systems and supports the EU's ongoing efforts to implement the AI Act, including the development of industry codes of practice.
Abstract
The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.