In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?

Previous claims about LLM self-control may have been overstated: A recent study claimed LLMs could control their own internal representations through neurofeedback, but this paper shows those results likely relied on "superficial mechanisms"—essentially, the models were responding to information visible in the prompt rather than demonstrating genuine internal control.

The "privileged access" requirement matters for rigorous testing: When researchers redesigned the experiment so that control targets were truly hidden from external observation (satisfying "privileged access"), the models failed to demonstrate reliable control, suggesting previous positive results were artifacts of flawed experimental design rather than evidence of real metacognitive ability.

Current LLMs lack demonstrated metacognitive abilities: The paper provides evidence that LLMs cannot reliably monitor and adjust their own internal mental states in the way humans can through neurofeedback, which has implications for how much we should trust these systems to self-regulate or self-correct.

AI safety assessments need stricter methodological standards: For evaluating whether AI systems have genuine self-awareness or self-control capabilities—important for safety considerations—researchers must use evaluation methods that prevent models from "cheating" by relying on surface-level pattern matching rather than true internal access.