Reproducibility is not construct validity: LLM measurement of institutionally situated communication

High reproducibility doesn't guarantee accuracy: The study found that LLM annotations of AI Act consultation texts achieved extremely high reproducibility scores (intraclass correlations > 0.99), yet failed to match what people actually reported in surveys about their concerns—demonstrating that consistent AI measurements can still measure the wrong thing.

Different groups say different things in different contexts: Business associations expressed significantly greater concern about AI risks in their written consultations than in their survey responses (effect size g = +1.0), while public authorities showed the opposite pattern, suggesting stakeholders strategically adjust their messaging depending on the communication channel.

Geography matters for AI policy positions: Stakeholders from neighboring European countries showed similar text-based stances on AI safety concerns (Moran's I = 0.347, p = 0.036), indicating that geographic proximity and regional factors influence how organizations publicly communicate about AI regulation.

Validation requires multiple checks, not just reproducibility: Organizations using LLMs to measure public opinion or stakeholder positions should validate that their measurements actually capture the intended concept, not just that the AI produces consistent results—reproducibility alone is insufficient for responsible AI deployment in policy contexts.

Survey responses remain reliable for predicting real-world positions: Despite divergences between text and survey measures, survey-reported concerns strongly predicted support for explainability requirements across all groups, suggesting traditional survey methods retain value alongside AI text analysis.