Is explainable AI a race against model complexity?

Explainability becomes harder as AI systems grow larger: The paper argues that as models approach "brain-scale" complexity, providing explanations for individual predictions may become practically impossible, suggesting we should prepare for a future where we cannot explain every decision an AI system makes.

Explanations are not objective or neutral: The authors note that explanations of AI behavior "cannot remain objective or apolitical," meaning the way we explain AI decisions will inevitably reflect human choices and values rather than being purely technical descriptions.

We can use AI systems even without full understanding: The paper challenges the assumption that we need complete explanations to benefit from AI, arguing that "models can be useful even when both model and explanation are incorrect"—suggesting practical utility doesn't require perfect transparency.

Our current understanding of how models work has limits: The authors contend that our "functionalist understanding" of AI models provides less advantage than commonly assumed, implying that even our best technical frameworks may not be sufficient to explain increasingly complex systems.

The explainability gap may be acceptable: Rather than viewing the inability to explain complex AI as a fundamental failure, the paper suggests "this is less problematic than it seems," proposing we may need to accept and work within the limits of explainability rather than treating it as a solvable problem.