Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

Policymakers face a critical timing problem: The paper highlights that AI policies must be created quickly, but their actual effectiveness may only become clear in the future—making it difficult to know if regulations will actually work as intended.

Transparency requirements have uneven protective effects: The study found that the EU AI Act's transparency policy (Article 50) was perceived as effective at reducing harms in labor and well-being impacts, but "largely ineffective" at addressing social cohesion and security risks, suggesting that one-size-fits-all regulations may not protect against all types of AI harms.

A new method can help test policies before implementation: The researchers developed a technique using AI to generate realistic scenarios showing what might happen with and without a specific policy, then had 234 people evaluate these scenarios—offering policymakers a way to stress-test regulations before they take effect.

Human perception of risk matters for policy evaluation: Rather than relying solely on technical analysis, the study measured how people actually perceive policy effectiveness across four dimensions (severity, plausibility, magnitude, and impact on vulnerable groups), recognizing that policies must address real-world concerns to be legitimate.

This method could improve policy iteration: By demonstrating how to systematically evaluate whether specific regulations mitigate specific harms, the paper provides a reusable framework for policymakers to refine AI regulations and identify which policies work best for which types of risks.