Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline

Synthetic data is now embedded throughout AI development, not just at one stage. Researchers and practitioners commonly use one large generative model to create synthetic data for training or evaluating other AI systems, fundamentally changing how AI workflows operate.

Synthetic data addresses real practical needs but creates new risks. While practitioners report that synthetic data is "crucial for addressing data scarcity" and makes large-scale evaluation of generative AI systems feasible, the paper identifies significant challenges: difficulty controlling auxiliary model outputs, struggles to accurately represent underrepresented groups in synthetic data, and inability to scale validation practices beyond manual inspection.

Current practices are outpacing responsible AI guidance. Despite widespread use of synthetic data across the industry, "policy guidance and best practices for its responsible use have not kept up with these rapidly evolving industry trends," leaving a gap between what companies are doing and what experts recommend.

Validation of synthetic data remains a bottleneck. Organizations struggle to validate synthetic data at scale, relying primarily on manual inspection—an approach that cannot keep pace with the volume of synthetic data now being generated and used in AI systems.

Representation gaps persist in synthetic data. The paper highlights that practitioners face particular challenges in "generating data that accurately depict underrepresented groups," suggesting that synthetic data may perpetuate or worsen existing fairness problems rather than solve them.