Explanations of Large Language Models Explain Language Representations in the Brain

XAI methods can reveal what drives AI-brain alignment: The study demonstrates that explainable AI techniques—specifically attribution methods that identify which input words most influence an LLM's predictions—successfully predict human brain activity during language processing, suggesting these explanations capture meaningful aspects of how both AI and brains process language.

Different layers of AI models align with different brain regions: Early layers of the LLM show greater sensitivity to word types and align with auditory processing regions, while final layers rely more on positional information and align broadly across the cortex, indicating that understanding AI decision-making requires examining internal structure, not just overall outputs.

AI explanations outperform raw model representations for brain prediction: Attribution-based explanations were more effective at predicting brain activity than the LLM's internal representations alone, particularly in early auditory regions, suggesting that how we explain AI decisions may be more informative than the decisions themselves.

Responsible AI development benefits from neuroscience insights: By connecting explainability methods to brain activity, this research provides a biological grounding for evaluating whether AI explanations are actually capturing meaningful computational processes, offering a new framework for assessing explanation quality beyond traditional metrics.