Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies

LLMs struggle with basic arithmetic despite advanced capabilities: Transformer-based large language models show surprisingly poor performance on elementary arithmetic problems, raising critical concerns about their reliability and safety for real-world deployment where accurate calculations matter.

LLMs learn arithmetic similarly to humans: The research found that LLMs exhibit learning patterns comparable to human learners, mastering simpler subtasks faster than complex ones—suggesting these models may share fundamental cognitive processes with human reasoning rather than operating as "black boxes."

Human learning strategies can improve AI performance: Applying problem-solving techniques and cognitive methods proven effective for human learners—such as task decomposition and strategic training approaches—significantly improved LLM accuracy on arithmetic, demonstrating that insights from educational psychology can enhance AI systems.

Explainability is achievable through decomposition: By breaking down arithmetic tasks into well-defined subtasks and analyzing learning patterns, researchers could illuminate how LLMs process information internally, making these systems more transparent and interpretable rather than remaining opaque decision-makers.

Responsible AI requires understanding fundamental limitations: This work highlights the importance of thoroughly testing AI systems on basic tasks before deployment, as gaps in elementary reasoning could have serious consequences for safety-critical applications and user trust.