Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

Most LLM alignment studies using synthetic personas lack transparency about who they're supposed to represent: Only 35% of the 63 peer-reviewed studies reviewed discussed whether their synthetic personas actually matched real populations, raising questions about whether findings apply to actual users.

Researchers often fail to clearly specify which real people and tasks their experiments are meant to study: The analysis found that "task and population of interest are often underspecified," meaning readers cannot determine if results about how an AI behaves with a particular persona are actually relevant to real-world use cases.

Current persona experiments focus on limited demographic characteristics: Most studies only represent a narrow range of sociodemographic attributes, potentially missing important differences in how AI systems interact with diverse user groups and potentially reinforcing biases.

The field needs standardized transparency practices: The authors propose a persona transparency checklist requiring researchers to clearly document their sampling methods, ground personas in real empirical data, and explain how well their experiments reflect actual usage conditions—practices that would help readers evaluate research credibility.

Better experimental rigor could improve AI safety research: By making persona-based experiments more representative and transparent, the field can produce more reliable evidence about how to align AI systems with human values across different populations, rather than potentially misleading findings based on unrealistic personas.