Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

Current explainability methods are inadequate for tracking how LLMs change: Traditional AI explainability approaches treat models as static objects, while comparing explanations across different model versions fails to explain why a behavioral shift occurred. The paper argues we need new methods that focus on the transition itself rather than just comparing two separate snapshots.

Regulatory compliance depends on understanding model modifications: The EU AI Act, US state legislation, and Chinese AI regulations all require documenting causal chains when AI systems undergo substantial modifications. Current explainability gaps create governance risks by making it impossible to demonstrate compliance with these requirements.

A new explainability framework (XAI_Δ) is needed: The authors propose "Comparative XAI" as a novel approach that treats behavioral shifts as the primary object of explanation, asking how and why specific interventions (scaling, fine-tuning, reinforcement learning, or in-context learning) transform a model's behavior.

Explanations must meet four key standards: Proposed explainers should satisfy comparability (showing what changed), validity (accurately reflecting the change), actionability (enabling stakeholders to respond), and monitoring (enabling ongoing oversight of model behavior).

This gap poses real governance risks: Without proper methods to explain behavioral shifts, organizations cannot adequately document or justify modifications to deployed AI systems, potentially violating emerging AI regulations across multiple jurisdictions.