Auditing large language models: a three-layered approach

Auditing alone cannot solve all LLM risks: The paper emphasizes that while auditing is a valuable governance tool, it has inherent limitations and should not be viewed as a complete solution to the ethical and social challenges posed by large language models.

A coordinated three-layer approach is needed: The authors propose that effective LLM governance requires audits at three complementary levels—examining the technology providers who create LLMs, the models themselves before release, and the specific applications built on top of these models—with each layer informing the others.

Current auditing methods are insufficient for LLMs: Existing audit procedures were not designed to handle the unique challenges LLMs present, particularly their "emergent capabilities" (unexpected abilities that arise during training) and their adaptability to many different downstream tasks, creating a governance gap that needs to be addressed.

Structured coordination between audit layers improves effectiveness: When audits are conducted in a systematic and coordinated manner across all three levels—rather than in isolation—they become more feasible and effective at identifying and managing ethical and social risks.

Auditing expands the governance toolkit but requires realistic expectations: The paper positions auditing as one important mechanism among several needed to ensure LLMs are designed and deployed ethically, legally, and robustly, rather than as a standalone solution.