AI Ethics Statements -- Analysis and lessons learnt from NeurIPS Broader Impact Statements

Ethics statements increased transparency but revealed uneven engagement: The NeurIPS 2020 requirement showed that while ethics statements can make researchers reflect on societal impacts, the quality and depth of these reflections varied significantly across geography, institution type, and research area, suggesting that mandatory statements alone don't guarantee meaningful ethical consideration.

Many researchers struggled to identify potential harms: The analysis found that researchers were often more willing to discuss benefits of their work than potential harms or negative applications, indicating a gap between the goal of balanced impact assessment and researchers' actual practices in identifying risks in their own work.

One-size-fits-all policies may not work across diverse research contexts: The study found substantial differences in how researchers from different regions, affiliations (academia vs. industry), and subfields approached impact statements, suggesting that governance mechanisms need to be tailored rather than uniformly applied to all research areas.

Checklist approaches replaced statements due to implementation challenges: The shift from mandatory impact statements (2020) to optional checklists (2021) reflects practical difficulties with the statement requirement, highlighting that well-intentioned ethics policies can face adoption barriers and may need iterative refinement based on real-world feedback.

Ethics governance requires ongoing learning and adaptation: The paper emphasizes that as AI governance mechanisms continue to evolve across conferences and institutions, evidence-based evaluation of what works—like this analysis of NeurIPS's experiment—is essential for designing more effective policies.