An Investigation of the NeurIPS and ICML 2025 Position Tracks

Position tracks are dominated by critique rather than new direction-setting: Three-quarters of audited submissions to the NeurIPS and ICML 2025 Position Tracks critique existing benchmarks, evaluations, or methodologies, rather than proposing new frameworks or tools that could reshape the field's research agenda.

Critique alone doesn't predict what reviewers value: While papers that couple their arguments to specific artifacts (benchmarks, methodologies) score highly on technical rigor measures, this "evidentiary depth does not predict reviewer rating," suggesting reviewers may be evaluating these papers inconsistently or by unstated criteria.

Agenda-shifting papers give the field new things to work with: Landmark papers that actually changed ML research (like AlexNet, the Transformer, and "Concrete Problems in AI Safety") typically introduced concrete artifacts—measurement protocols, benchmark proposals, datasets, audit templates, or falsifiable experimental programs—rather than just identifying problems with existing work.

The track's design may inadvertently narrow what counts as rigorous contribution: By hosting critiques well but not explicitly soliciting direction-setting work, the position tracks risk shaping what kinds of research claims become "legible to reviewers" and what forms of evidence count as rigorous, potentially limiting the field's ability to imagine fundamentally different approaches to responsible AI.

Targeted CFP changes could broaden submissions without losing existing value: The authors propose four call-for-papers interventions to attract more direction-setting work alongside existing critiques, suggesting the problem is addressable through venue-level design rather than requiring wholesale changes to how the community evaluates research.