Tell Me Why: Designing an Explainable LLM-based Dialogue System for Student Problem Behavior Diagnosis

Explanations increase teacher trust in AI systems: In a user study with 22 pre-service teachers, those who received natural-language explanations for the system's recommendations reported significantly higher trust compared to those without explanations, demonstrating that transparency directly impacts user confidence in AI-assisted decision-making.

AI can support complex educational diagnosis: The system successfully helps teachers synthesize multiple types of information to diagnose student behavioral problems and plan interventions through multi-turn dialogue, suggesting LLMs can handle nuanced educational tasks that require synthesizing diverse evidence.

Current AI systems lack transparency by default: The research identifies a critical gap where fine-tuned LLMs can make recommendations but "rarely explain why," highlighting that capability alone is insufficient—explainability must be deliberately designed into educational AI systems.

Technical methods can identify supporting evidence: The hierarchical attribution method outperformed baseline approaches at identifying which parts of the dialogue conversation actually supported each recommendation, showing that explainable AI techniques can pinpoint the reasoning behind system outputs in meaningful ways.

Explainability is essential for responsible AI in education: By connecting technical explainability improvements to increased user trust among educators, this work demonstrates that responsible AI deployment in schools requires not just accurate recommendations, but transparent reasoning that teachers can understand and evaluate.