Math Education Digital Shadows for Investigating Learning with GenAI: Mathematics Performance, Anxiety, and Confidence in LLMs

LLMs exhibit human-like biases in mathematics: The study found that large language models display negative attitudes toward mathematics, logical fallacies, and overconfidence in their mathematical abilities—mirroring psychological patterns seen in human learners, which has implications for how these models might influence student attitudes if deployed as tutors.

AI behavior changes based on how it's prompted: The research revealed significant differences in how LLMs performed and responded depending on whether they were prompted as humans versus AI assistants, suggesting that the framing and context of AI interactions can substantially affect their outputs and reasoning patterns.

Overconfidence is a key concern: Across the 14 different LLMs tested, the models demonstrated overconfidence in their mathematical answers, meaning they often provided incorrect solutions while expressing high certainty—a particularly problematic trait for educational applications where students might trust incorrect explanations.

Comprehensive dataset enables safer AI development: The MEDS dataset of 28,000 test runs provides learning scientists and AI developers with concrete data about LLM mathematical reasoning, biases, and confidence patterns, offering a foundation for building more reliable and trustworthy AI tutoring systems.

Performance varies significantly across different LLM models: Testing 14 different LLMs (including Mistral, Qwen, DeepSeek, and others) revealed meaningful differences in mathematical performance and psychological characteristics, indicating that not all AI models are equally suitable for educational purposes.