Finding Words Associated with DIF: Predicting Differential Item Functioning using LLMs and Explainable AI

DIF prediction models showed modest but meaningful performance: The researchers fine-tuned multiple AI language models to predict differential item functioning (DIF)—when test questions perform differently for different groups of students—achieving prediction accuracy (R²) ranging from .04 to .32 across different student groups, suggesting the models captured real patterns in item bias.

AI identified specific words linked to unfair test performance: Using explainable AI methods, the team pinpointed particular words in test questions associated with DIF, providing concrete, actionable targets for item writers and reviewers rather than vague concerns about bias.

Many flagged words reflect intentional test design, not bias: A key finding was that many words the AI associated with DIF actually reflected minor sub-topics intentionally included in the test blueprint, not construct-irrelevant content that unfairly disadvantages certain students—explaining why traditional manual reviews of biased items often produce unclear results.

This approach could democratize fair assessment: The method offers promise for improving test fairness in under-resourced assessment programs and for smaller student populations where traditional statistical bias detection methods don't work due to insufficient sample sizes.

Practical applications span item development and review: The AI screening tool can be used proactively during item writing for immediate revision, or retrospectively to help interpret results from traditional bias analyses by highlighting key problematic words.