The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review

Data quality is foundational to trustworthy medical AI: The paper emphasizes that data quality directly dictates how machine learning products behave in medicine, making systematic evaluation of training and test data essential for regulatory approval of medical AI applications.

The METRIC-framework provides 15 specific dimensions for assessment: Rather than vague standards, the researchers synthesized existing knowledge into a concrete framework with 15 awareness dimensions that developers should investigate when evaluating medical datasets, offering practical guidance for building trustworthy AI systems.

Poor data quality is a major source of bias and unfairness: The paper identifies data quality issues as a primary driver of biases in medical AI, which can lead to unfair or discriminatory outcomes for patients—highlighting why rigorous data assessment is critical before deployment.

Systematic data evaluation can accelerate responsible AI approval: By incorporating structured data quality assessment into regulatory processes, the framework has potential to both speed up approval of legitimate medical ML products and prevent problematic ones from reaching patients.

Current medical AI development lacks standardized data quality practices: The systematic review of 2,362 studies found only 62 met eligibility criteria, suggesting that standardized approaches to evaluating medical training data are not yet widely adopted in the field, indicating a significant gap in current practice.