What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes

Polysemanticity (neurons responding to multiple unrelated features) may be harder to avoid than previously thought. The paper challenges the conventional explanation that polysemanticity only occurs when there aren't enough neurons for all features in the data. Instead, researchers found that polysemanticity can arise "incidentally" even when ample neurons are available, suggesting it's a more fundamental challenge to AI interpretability than the field previously understood.

Random chance during neural network initialization can lock in polysemanticity. The researchers demonstrated that when a network is randomly initialized, multiple unrelated features may coincidentally be assigned to the same neuron by chance alone. During training, these accidental overlaps then get reinforced and strengthened, meaning polysemanticity can persist even without any computational pressure to compress information.

Common training practices may inadvertently promote polysemanticity. The paper identifies regularization techniques and neural noise as factors that can increase incidental polysemanticity. This is significant for AI safety because it means that standard methods used to improve neural network performance might simultaneously make networks harder to interpret and understand.

The field needs better understanding of the performance-interpretability tradeoff. The authors call for more research to quantify how much polysemanticity is truly unavoidable versus how much could be eliminated through different training approaches. This is crucial for determining whether interpretability improvements are realistically achievable without sacrificing model performance.