When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretability

Feature steering is fragile and unreliable: While researchers can extract interpretable features from AI models and attempt to control them, the paper finds this control is highly sensitive to small changes like which layer you target, how strongly you steer, and the surrounding context—raising serious concerns about using this method for safety-critical applications.

Current mechanistic interpretability may be oversold: The paper successfully reproduces some results from Anthropic's influential work but warns that demonstrations work well in "selected cases" while often failing to achieve the "systematic reliability required for safety-critical applications," suggesting the field should be more cautious about its current capabilities.

Similar features are hard to distinguish: The researchers found it difficult to tell thematically similar features apart from one another, meaning the approach may create a false sense of understanding when features that look different are actually quite similar—a critical problem if these tools are meant to help humans oversee AI systems.

A shift in focus is needed for AI safety: Rather than continuing to prioritize understanding what happens inside AI models, the paper argues the field should focus more on reliably predicting and controlling what models actually output—a more practical approach to ensuring AI systems behave as intended.