From Black Box to Transparency: Enhancing Automated Interpreting Assessment with Explainable AI in College Classrooms

Explainability matters more than raw accuracy: The researchers deliberately chose transparent, understandable features over "black box" machine learning models, demonstrating that educators and students benefit more from knowing why an AI system made an assessment decision than from simply receiving a score.

Different language skills require different measurement approaches: The study found that assessing interpretation quality requires multiple specialized metrics—translation accuracy (BLEURT and CometKiwi scores) for fidelity, pause patterns for fluency, and Chinese-specific phrase diversity for language use—rather than a single one-size-fits-all measurement.

AI can provide personalized learning feedback at scale: By making model predictions interpretable through Shapley Value analysis, the system can offer detailed diagnostic feedback to individual students, supporting self-directed learning in ways that automated scores alone cannot achieve.

Data quality and transparency are foundational: The researchers addressed common AI pitfalls including data scarcity and imbalance through careful feature engineering and data augmentation, highlighting that responsible AI systems require thoughtful data preparation, not just sophisticated algorithms.

Transparent AI can replace human evaluation reliably: The framework demonstrates "strong predictive performance" as a scalable alternative to traditional human grading, suggesting that explainable AI systems can handle high-volume educational assessment while maintaining the transparency needed for institutional trust and student understanding.