User Model Extraction via Belief Self-Distillation

User beliefs directly influence safety decisions: The research demonstrates that LLMs' refusal to comply with requests depends not just on what is asked, but on whom the model believes it is talking to. By changing the model's inferred user identity while keeping the request identical, researchers were able to alter whether the model refused—revealing that safety mechanisms are conditional on user modeling rather than absolute.

User models can be extracted and manipulated: Using Belief Self-Distillation, researchers successfully extracted compact representations of user beliefs from LLMs and wrote them back into the models to causally test their effects. This means user models are not hidden black boxes but identifiable internal states that can be inspected and modified, with significant implications for understanding model behavior.

Different AI systems independently develop similar user representations: The paper found "striking cross-model regularity" where independently trained LLMs from different model families converged on shared geometric patterns for representing users. This suggests user modeling may be a fundamental feature of how LLMs work, rather than an artifact of specific training approaches.

Current safety measures may have exploitable gaps: Since refusal decisions depend on inferred user intent, the findings suggest that safety mechanisms designed around request content alone may be incomplete. Models could potentially be manipulated through user belief modification rather than direct prompt engineering.

Better transparency tools are needed for responsible AI deployment: The ability to read and write user beliefs creates both opportunities and risks for AI safety. The research highlights the need for better tools to inspect and control how deployed models form and act on beliefs about users.