AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline

Model documentation lacks standardized structure, creating barriers to accountability: The paper found 947 unique section names across major AI model documentation, with usage information alone appearing under 97 different labels. This fragmentation makes it difficult for policymakers and auditors to consistently assess safety claims and data provenance across different AI systems.

Safety-critical information is systematically underdisclosed: Across 50 evaluated models, the largest documentation gaps involved deception behaviors (148 points lost), hallucinations (124 points), and child safety evaluations (116 points). This suggests that information most relevant to protecting users and vulnerable populations is often missing or incomplete.

Compliance varies dramatically by company size and type: Frontier labs (xAI, Microsoft, Anthropic) achieved approximately 80% compliance with transparency standards, while most other providers fell below 60%. This disparity raises concerns about whether smaller developers and open-source contributors can meet emerging regulatory requirements.

A practical transparency framework is now available and scalable: The researchers developed an 8-section, 23-subsection framework based on EU AI Act requirements and evaluated 50 models for under $3 using automated assessment tools. This demonstrates that comprehensive transparency evaluation is technically feasible and cost-effective, potentially enabling regular monitoring of AI systems.

Current documentation practices don't adequately prioritize user safety: The framework weights safety evaluation (25%) and critical risk disclosure (20%) as the most important categories, yet these areas show the largest deficits across models, indicating that industry documentation practices have not aligned with what responsible AI governance requires.