EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners

Long-term tutoring performance differs from single-session evaluation: The benchmark found that "almost no combination sustains good tutoring over the full horizon" of 30 days, revealing that AI tutors performing well in isolated interactions may fail to maintain educational effectiveness over sustained relationships with learners.

Both the AI model and the tutoring system design matter equally: The research demonstrates that "tutoring quality belongs to the base model and the agent harness together rather than either alone," suggesting that responsible deployment requires attention to both the underlying LLM and how it's integrated into the educational system.

Simulated learners can realistically represent actual student learning: The benchmark's simulated learner was validated through a calibration check (ECE=0.049) and confirmed by a live-classroom field study, indicating that AI-powered educational evaluation can use realistic learning models grounded in real student data to assess tutor performance.

Multiple evaluation dimensions are necessary for responsible AI tutoring: The benchmark assesses tutors across five dimensions—learning gain, responsiveness, helpfulness, and two curriculum-design principles (Gagné and Rosenshine)—rather than a single metric, reflecting that responsible educational AI requires multi-faceted evaluation beyond just test scores.

Current AI tutoring systems need improvement for real-world deployment: The finding that almost no agent-model combination sustained good tutoring performance over 30 days suggests that existing LLM-based tutoring systems require significant refinement before they can reliably support long-term student learning.