EthicsFairnessAccountabilitycs.AI
Measuring Ethics in AI with AI: A Methodology and Dataset Construction
Pedro H. C. Avelar, Rafael B. Audibert, Anderson R. Tavares, Luís C. Lamb
July 26, 2021
Brazilian Conference on Intelligent Systems
4 citations
Key Takeaways
-
AI can help measure itself: The paper demonstrates that machine learning models can be trained to automatically identify and classify ethical issues in AI research papers, offering a scalable way to track how much attention the AI community is paying to ethics—rather than relying solely on manual review.
-
Expert curation matters for measurement: The researchers built their classification system using a manually curated dataset created by experts, highlighting that accurate AI measurement requires human expertise and careful groundwork, not just algorithmic automation.
-
Metrics are tools for accountability: The paper emphasizes that developing sound metrics for AI ethics serves a practical purpose: helping decision-makers (in academia, government, and industry) understand and evaluate the real-world impacts of AI advances, contributing to more trustworthy and fair AI systems.
-
Measurement is becoming a priority across sectors: The authors note that multiple influential field reports and policy documents now focus on measuring AI phenomena, suggesting that stakeholders recognize measurement as essential for responsible AI development and governance.
Abstract
Recently, the use of sound measures and metrics in Artificial Intelligence has become the subject of interest of academia, government, and industry. Efforts towards measuring different phenomena have gained traction in the AI community, as illustrated by the publication of several influential field reports and policy documents. These metrics are designed to help decision takers to inform themselves about the fast-moving and impacting influences of key advances in Artificial Intelligence in general and Machine Learning in particular. In this paper we propose to use such newfound capabilities of AI technologies to augment our AI measuring capabilities. We do so by training a model to classify publications related to ethical issues and concerns. In our methodology we use an expert, manually curated dataset as the training set and then evaluate a large set of research papers. Finally, we highlight the implications of AI metrics, in particular their contribution towards developing trustful and fair AI-based tools and technologies. Keywords: AI Ethics; AI Fairness; AI Measurement. Ethics in Computer Science.