In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?
cs.AI
Submitted: 2026-09-01
Updated: 2026-09-01
Code: https://github.com/AokiKoshiro/in-context-neurofeedback
License: http://creativecommons.org/licenses/by/4.0/
The gist: Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety.
Terminology
Abstract
Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety. A recent study applied neurofeedback to LLMs and claimed that they can control their internal representations. However, the reported control may rely on superficial mechanisms rather than genuine internal access because the control targets in that study are not privileged, meaning that a third party can infer them from the prompt. We redesign the neurofeedback paradigm for LLMs so that the control target satisfies the privileged access requirement, which is closer to neurofeedback experiments in human cognitive neuroscience. Under this stricter setting, the models do not demonstrate reliable control over privileged internal representations, suggesting that previously reported control cannot exclude the possibility that it relies on superficial mechanisms. Our results indicate that rigorous assessments of metacognition in LLMs require evaluation methods that demand privileged access.
Sources
- Does It Make Sense to Speak of Introspection in Large Language Models?
- The Llama 3 Herd of Models
- Detecting High-Stakes Interactions with Activation Probes
- Language Models (Mostly) Know What They Know
- Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- Training Language Models to Explain Their Own Computations
- Privileged Self-Access Matters for Introspection in AI
- Steering Language Models With Activation Engineering
- Qwen3 Technical Report
- Reasoning Models Struggle to Control their Chains of Thought
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems