Human or AI? Comparing Design Thinking Assessments by Teaching Assistants and Bots

AI showed limited reliability in assessing creative and subjective dimensions: The study found "low statistical agreement between instructor and AI scores for empathy and pain points," suggesting AI struggles with evaluating abstract creative concepts that require contextual understanding—a critical concern for educators considering AI-assisted grading in design fields.

Teachers overwhelmingly preferred human assessment: In the study of ten sample assessments, "teachers preferred TA-assigned scores in six of ten samples," indicating that despite potential efficiency gains, educators still trust human judgment more for evaluating creative work, particularly when nuanced feedback matters.

AI performed better on objective visual elements than subjective concepts: The research found "slightly higher alignment for visual communication" compared to empathy and pain points assessment, suggesting AI may be more reliable for technical or structural evaluation but less suitable for assessing creative thinking and human-centered design principles.

Hybrid models show promise but require careful design: Rather than replacing human assessors, the study "underscores the need for hybrid assessment models that integrate computational efficiency with human insights," suggesting the most responsible approach combines AI's consistency and speed with teachers' ability to capture contextual nuance.

Educators value AI for feedback generation, not final grades: Qualitative feedback highlighted "the potential of AI for formative feedback, consistency, and student self-reflection," indicating AI may be most responsibly deployed to support learning rather than make final assessment decisions.