Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor

Narrow focus on mathematical correctness is insufficient: The paper argues that AI research has traditionally defined rigor only through proper application of mathematical, statistical, and computational methods. This limited view has allowed problematic AI work to be published and deployed despite being "methodologically rigorous," contributing to overblown claims about AI capabilities.

Six dimensions of rigor are needed for responsible AI: Beyond methodology, rigorous AI work should include: understanding what background knowledge justifies the research direction (epistemic rigor), recognizing how personal and community biases influence decisions (normative rigor), clearly defining what concepts are being studied (conceptual rigor), being transparent about what findings are reported (reporting rigor), and ensuring conclusions are actually supported by evidence (interpretative rigor).

Current practices enable irresponsible claims: The paper suggests that focusing only on methodological rigor while neglecting the other five dimensions has allowed researchers to make exaggerated claims about AI systems' capabilities without adequate scrutiny, contributing to public misunderstanding and potentially harmful deployment decisions.

A shared framework improves accountability: By providing language and structure around these six dimensions of rigor, the paper aims to enable better dialogue between AI researchers, policymakers, journalists, and other stakeholders—making it easier to identify and challenge work that appears rigorous on the surface but lacks depth in other critical areas.