FairnessAccountabilityTransparencycs.CY
Where Assessment Validation and Responsible AI Meet
Jill Burstein, Geoffrey T. LaFlair
November 4, 2024
arXiv.org
5 citations
1 influential
Key Takeaways
-
Validity, reliability, and fairness are foundational ethical principles that have long been central to educational and psychological testing standards, and these same principles are now essential for ensuring responsible AI use in assessment contexts, particularly for high-stakes decisions like automated scoring of student work.
-
A unified framework combining classical assessment validation with responsible AI principles is necessary to ensure that AI systems used in educational and psychological testing maintain human values, enable proper oversight, and support broader social responsibility—moving beyond treating these as separate concerns.
-
Generative AI advances have created new policy and implementation challenges that require updated guidance on how to apply responsible AI principles specifically to assessment contexts, suggesting that existing standards alone are insufficient for newer AI technologies.
-
Assessment validation theory and responsible AI ethics share common ground in their emphasis on fairness and ethical use, meaning that professionals implementing AI in testing can draw on decades of assessment expertise while also adopting contemporary AI ethics practices to protect test-takers and maintain the integrity of educational and psychological decisions.
Abstract
Validity, reliability, and fairness are core ethical principles embedded in classical argument-based assessment validation theory. These principles are also central to the Standards for Educational and Psychological Testing (2014) which recommended best practices for early applications of artificial intelligence (AI) in high-stakes assessments for automated scoring of written and spoken responses. Responsible AI (RAI) principles and practices set forth by the AI ethics community are critical to ensure the ethical use of AI across various industry domains. Advances in generative AI have led to new policies as well as guidance about the implementation of RAI principles for assessments using AI. Building on Chapelle's foundational validity argument work to address the application of assessment validation theory for technology-based assessment, we propose a unified assessment framework that considers classical test validation theory and assessment-specific and domain-agnostic RAI principles and practice. The framework addresses responsible AI use for assessment that supports validity arguments, alignment with AI ethics to maintain human values and oversight, and broader social responsibility associated with AI use.