Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

arXiv:2610.20767 · cs.RO · Submitted 2026-10-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models".

Dev: The gist The paper investigates whether recovering multi-dimensional feature structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn How it works The…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So, we’re looking at this paper, "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models." It looks like they are digging into how we can actually understand what's happening inside these large vision language action models.

Dev: Yeah, it seems to be about finding a way to get an interpretable interface for steering these models and shaping the data they learn from. It’s trying to figure out which internal features correspond to perception, language, robot state, and so on.

Taro: It sounds like they are trying to build a better map of the model's computation so we can actually control it more reliably when it messes up in the real world.

Rosa: Exactly. They use something called block-sparse featurizers, or BSFs, to recover these multi-dimensional features under the hood of a vision language action model. It’s not just one feature; they are recovering things like how the robot perceives its surroundings, what language is being used, and even its proprioceptive state and episode phase.

Dev: And they are doing this by treating each feature as living in a low-dimensional subspace. This helps them find features that actually capture the geometry of the problem better than standard methods.

Taro: I'm interested in how they validate these features. They aren't just guessing what the features are; they’re causally validating them through fine-grained steering experiments where they change perception or semantic representation and see exactly how that predicts changes in downstream action prediction.

Rosa: And it gets deeper than just validation. They use feature-based data attribution to trace specific behaviors back to the training data. If you want to know why a robot fails in a certain way, you can find which few tokens in the training set are responsible for that failure, and masking them removes the behavior.

Dev: That’s powerful for data curation. They found they only needed to mask about one point four percent of fine-tuning tokens to change the robot's behavior precisely <ref:2610.20767#pg3,1.4% of fine-tuning tokens>. That level of control over the training data is significant for safety checks, right?

Taro: It makes sense because if you know exactly which part of the learned representation causes a failure mode, you can target that specific piece of data. We can use this feature-guided data masking to selectively delay failure recovery, which is pretty smart.

Title and authors: Rosa: And they compared BSFs against Sparse Autoencoders, or SAEs, and found that the BSFs are better at capturing the computational geometry qualitatively and quantitatively. They argue that because BSFs have an inductive prior of low-dimensional features that are smooth over time, they are a more natural fit for these complex models than traditional SAEs.

Dev: The comparison was interesting because they trained those SAE baselines on much larger activation corpora, about two orders of magnitude more than the pooled activations in the next largest dataset from Swann et al. two thousand twenty-six <ref:2610.20767#pg2>. Still, the BSFs still showed superior performance when it came to recovering and intervening on task-relevant structure.

Taro: It’s a big deal if these feature geometries truly represent the underlying concepts of perception, language, and control in a unified way. That’s what they show in Figure one where you see end-effector position, fruit and vegetable semantics, and proprioceptive state all being recovered by BSF blocks <ref:2610.20767#pg1>.

Rosa: They break down the features into categories: visual features for objects and the robot itself, visual language features that show organization consistent with a shared vision language representation between sight and language, and then task state variables like gripper configuration or end-effector Cartesian coordinates over a single episode.

Dev: That’s a lot of information packed into these recovered features. But the steering experiments showed something even more practical about how useful these are for action control. Geometric steering proved to be the most compact, with just one BSF block achieving ninety-nine point seven percent recovery of the behavior needed for redirection tasks.

Taro: I was looking at that result on closed-loop target-object redirection. They found that BSF features redirect the policy more reliably with fewer features than SAEs, selecting the redirected object in every trial and completing the task in ninety-three point three percent of trials using about two point three features on average, compared to two point five SAE features. That's a solid reduction in complexity for control tasks.

Rosa: It really shows that matching featurizers to representation geometry gives you more effective tools for understanding and changing robot behavior than just using standard linear representations or traditional sparse autoencoders alone. The paper "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models" is showing us how to probe the internal structure of these models.

Title and authors: Dev: Overall, the implication here is that we can use these recovered features to build targeted interventions in policy behavior, and we can use feature attribution to clean up the training data. It gives us a way to trace failures directly back to specific representation components.

Taro: If this works reliably outside of the lab, like Rosa wants us to test, it means we have a better handle on what causes those kinds of unexpected misbehaves when the world doesn't behave exactly as expected.

Rosa: It suggests that future systems will need these tools to ensure they are safe and robust when deployed in physical environments. We’re moving toward models that can do complex things, and understanding the internal variables is essential for building trust in those deployments.

Dev: And we have to remember, though, the paper itself points out a limitation: BSFs are not uniformly superior; they found that SAEs perform better on some quantitative metrics when you look at activations from substantially larger unpooled activation corpora. So it’s not a one-size-fits-all solution for every deployment scenario.

Taro: That’s important context. It tells us that while BSFs are great for steering and understanding geometry, we still need to be careful about which featurizer we pick based on the specific dataset size or task complexity you're dealing with.

Rosa: So, to wrap up this discussion on "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models," the core finding is that recovering these multi-dimensional features provides an interpretable interface for steering vision language action models.

Dev: And the practical application is using those features to shape behavior through targeted interventions, and using data attribution to isolate where specific behaviors arise in the training dataset. It’s a tool for both understanding and controlling robot learning.

Taro: For us, it means we can start designing failure-informed learning systems that don't just react blindly, but know which representation component is causing the issue.

Rosa: Yeah, this work lays a foundation for how we can build more controllable and transparent foundation models for robotics by linking internal representation geometry directly to actionable control mechanisms.

Dev: We’re going to keep an eye on how these feature-based tools integrate into real-time control loops, which is where the safety case really gets tested.

Taro: And we’ll be watching those steering experiments closely to see how compact and effective a single BSF block actually is when we put it under pressure.

The paper's summary: Rosa: So, to wrap up, this paper shows that we can recover multi-dimensional features inside those vision language action models using block-sparse featurizers, or BSFs. These features let us actually steer the robot's behavior and even shape the data it learns from.

Dev: Yeah, it’s about finding an interpretable interface for steering these models and controlling what they learn. They aren't just looking at one single feature; they are recovering things like how the robot perceives things, what language is involved, its physical state, and even which part of the episode it’s in.

Rosa: And they do this by treating each feature as living in a low-dimensional subspace. This helps them find features that capture the actual geometry of the problem better than some other methods we’ve seen before.

Dev: They causally validated these features through steering experiments where they intentionally change how the AI perceives something and see exactly how that predicts changes in its actions. It shows a clear link between those internal features and external behavior.

Rosa: And they used feature-based data attribution to trace specific behaviors back to the training data itself. They found that you only need to mask about one point four percent of the training tokens to remove a specific failure mode, which is super precise for data curation.

Dev: That level of control over the training set is important for safety checks because you can target exactly what caused a failure in the first place. It’s not just about knowing *that* it failed, but knowing *why* and where in the training data that reason came from.

Rosa: They compared these BSFs against sparse autoencoders, or SAEs, and found that the BSFs are better at capturing this underlying structure qualitatively and quantitatively. They argue that because BSFs have a prior of low-dimensional features that are smooth over time, they fit these models' geometry much better than traditional SAEs.

Dev: The comparison was interesting because they trained those SAE baselines on activations from much larger datasets, about two orders of magnitude bigger than what we usually use for our baseline comparisons. Still, the BSFs still showed superior performance when it came to recovering and intervening on task-relevant structure.

Rosa: They break down the recovered features into categories: visual features for objects and the robot itself, language features that show how sight and language are aligned, and then variables about the task state like gripper configuration or coordinates over a single episode.

Dev: That’s a lot of information packed into those features. But they also found that geometric steering is the most compact way to redirect a task, with just one BSF block achieving nearly one hundred percent recovery of the required behavior for redirection tasks.

Rosa: It really shows that matching these featurizers to the representation geometry gives you more effective tools for understanding and changing robot behavior than just using standard linear representations alone.

Dev: And they found specific features, like F1515 in layer nine of a model, that activate during recovery from failed grasps. They show you exactly which part of the network is involved in that failure mode.

Rosa: So, these recovered features give us a way to probe the internal structure of these models and use that understanding to control them and clean up their learning data. What happens next is how we actually use this steering capability in real-time on physical systems, which we'll talk about next.

The paper's improvements: Taro: So, we've seen how BSFs help us steer and shape data using these multi-dimensional features in vision language action models, right? Now let’s talk about what the authors suggest we do next to make this even more useful for real robots.

Rosa: They point out that while BSFs are good at capturing the geometry, they aren't perfect all the time. So one improvement is moving towards a system that can "select features which steer robot behavior to target object on a set of discovery inputs and then evaluated, on held-out inputs with fresh noise seeds."

Dev: That sounds like an iterative safety check. It means you don't just test the model once; you constantly probe it with new scenarios and different noise levels to see if that specific feature actually steers the behavior reliably across different situations.

Taro: And they want to apply this idea of selective masking even more finely for failure recovery. They suggest masking tokens where a specific feature, like F1515, fires during a failed grasp, which lets you selectively delay the failure recovery without destroying all the useful information.

Rosa: That’s interesting because it moves beyond just general feature extraction into active intervention. It means we can fine-tune the data or the model's behavior right at the point where a known problem is about to happen.

Dev: From a controls standpoint, that implies a way to manage latency better. If you can selectively mask tokens in real-time based on an internal feature firing, you could potentially intervene much faster than waiting for a full policy retraining.

Taro: And they also emphasize that geometric steering is the most compact. They suggest focusing our efforts on finding those minimal sets of features—like just one BSF block—to achieve high success rates in redirection tasks.

Rosa: So the goal is to make these tools less about brute-force feature extraction and more about finding the most efficient, compact way to control the model using just a few key features.

Dev: That makes sense for deployment. A simpler, smaller set of features that reliably steers you in critical situations is always better than a complex one with twenty features that might only work on paper.

Taro: It’s about creating these highly compact, interpretable control signals that we can trust when the robot is interacting with the physical world and things get messy.

Rosa: And they also acknowledge a limitation: BSFs aren't universally better; they found that sparse autoencoders perform better on some quantitative metrics when you look at activations from substantially larger unpooled activation corpora.

Dev: That’s the caveat. So, the paper suggests we need to be careful about which featurizer we pick based on how much training data you have and what kind of performance metric matters most.

Taro: It means this isn't a single solution for every robot task; it depends on the representation geometry and the scale of the model you’re working with.

Rosa: Exactly. So, to summarize these improvements, they are pushing for more active intervention strategies and focusing on finding those minimal steering sets that give us reliable control.

Dev: It shifts the focus from just describing what's in the model to actively using those features to guide its actions under uncertainty.

Taro: And that leads us right into how we put this into practice, specifically how we can use these steering experiments to build robust, real-world systems that don't just learn but adapt gracefully when things go wrong.

Conclusion: Rosa: So, to wrap up this whole discussion on "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models," we’ve seen how BSFs allow us to recover internal features that give us a direct way to control robot behavior.

Dev: It really shows that matching these featurizers to the representation geometry gives you more effective tools for understanding and changing robot behavior.

Taro: I just want to emphasize how useful it is for failure modes. Because we can isolate exactly which feature is causing a problem, we can use that knowledge to build systems that recover much more gracefully in the real world.

Rosa: Right, and they also showed us how you can use this feature-based data attribution to trace those failures back to the specific training data tokens, which is huge for cleaning up datasets before we train new models.

Dev: It’s a solid way to ground our safety case because we’re not just relying on general performance numbers; we’re looking at the actual representation geometry that drives the behavior.

Taro: So, if you're building an autonomous system, you want to think about these steering mechanisms as a way to inject specific corrective signals into the policy loop when things deviate from what was expected.

Rosa: Exactly. And they did flag one thing: BSFs aren't uniformly superior; sparse autoencoders still perform better on some quantitative metrics if you use massive, unpooled activation corpora.

Dev: That’s a key caveat for engineers to remember: the best featurizer depends heavily on the data scale and what exactly you need to measure in terms of performance versus interpretability.

Taro: So, we can’t just pick one tool and assume it solves every autonomy problem; we have to match the feature geometry to the specific task structure.

Rosa: That’s right. And this whole study on "Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models" gives us a much clearer picture of how to build more controllable and transparent foundation models for robotics.

Dev: It moves us closer to systems where we can diagnose why a robot failed, not just that it failed, which is essential for deployment safety.

Taro: I’m really looking forward to seeing how these feature-guided interventions actually translate into robust, real-time control when the world throws unexpected surprises at the robot.

Aiden Swann, Mozes Jacobs, Mathilde Papillon, Siddharth Boppana, David Chanin, Dron Hazra, Usha Bhalla, Curt Tigges, Mac Schwagera Owen Lewis, Jack Merullo, Öekdeep Singh Lubana–Ekdeep Singh Lubana–Atticus Geiger, Ähomas Fel†▷Matthew Kowal†

Stanford University · Harvard University · University of California, Santa Barbara

cs.RO

Submitted: 2026-10-14

Updated: 2026-10-14

Code: https://github.com/goodfire-ai/robotics_multi_dimensional

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The paper investigates whether recovering multi-dimensional feature structure provides an interpretable interface for steering vision-language-action models and shaping the data from which

Key concepts

Block-Sparse Featurizers (BSFs): These are a method used to represent complex model activations by projecting them into low-dimensional subspaces, grouping related features into blocks. This structure helps recover the underlying multi-dimensional concepts that drive a robot's behavior, such as visual perception or control states.
Fine-Grained Steering: This involves making small, targeted changes to specific model features—like visual perception or semantic representation—to observe predictable and controlled changes in the resulting robot action. It proves which internal features are responsible for specific behaviors.
Feature-Based Data Attribution: This technique uses the identified BSF features to pinpoint exactly which parts of a training dataset cause a particular behavior, like failure recovery. By masking only a small percentage of tokens associated with that feature, researchers can selectively remove the unwanted behavior from the data.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Steering — 1× in this paper · explained in 2 episodes

Summary

The gist The paper investigates whether recovering multi-dimensional feature structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn

How it works

The core methodology involves investigating block-sparse featurizers (BSFs) to recover multi-dimensional task-relevant features underlying the model’s computation, such as perception, language, proprioceptive state, episode phase, and control variables BSFs represent each feature in a low-dimensional subspace The investigation causally validates these features through fine-grained steering experiments where targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction Furthermore, the study uses feature-based data attribution to isolate where a specific behavior arises in the training dataset by identifying fewer than 1.4% of training tokens whose masking is sufficient to remove the targeted behavior

Comparison of Featurizers

The researchers systematically compare BSFs and Sparse Autoencoders (SAEs) in vision-language-action (VLA) models, hypothesizing that BSFs, due to their inductive prior of low-dimensional features that are smooth over time, are a more natural fit than traditional SAEs They find that BSFs recover features spanning perception, language, proprioception, task state and episode progress better capturing computational geometry qualitatively and quantitatively When compared against SAE baselines trained on substantially larger, unpooled activation corpora—approximately two orders of magnitude more than the pooled activations in the next largest dataset from Swann et al. (2026)—the BSFs show superior performance in recovering and intervening on task-relevant structure

Causal Validation through Steering

The paper performs causal validation through fine-grained steering experiments across multiple layers and computational regimes, inducing targeted changes at several stages of the policy: what is perceived in the observation, which object is selected, where that object is localized Open-loop steering results show that a useful featurizer should enable targeted changes to policy behavior through a small number of feature edits that generalize to new scenes For instance, geometric steering proves most compact, with a single BSF block achieving 99.7% recovery Closed-loop target-object redirection demonstrates that BSF features redirect the policy more reliably with fewer features than SAEs, selecting the redirected object in every trial and completing the task in 93.3% of trials using 2.3 features on average compared to 2.5 SAE features

Feature Phenomenology and Application

The discovered BSF features are associated with visual, visual-language, and robot state and control concepts Visual Features include those selective for objects in the environment and for the robot itself, while Visual-Language Features show organization consistent with a shared vision–language representation that retains distinctions between members of a semantic category Task and Control Variables are represented by features such as gripper configuration and end-effector Cartesian coordinates over a single episode In terms of behavior shaping, the study identifies F1515, a three-dimensional BSF feature in layer nine of π0.5 that activates during recovery from failed grasps By masking the loss for any 16-step action chunk overlapping an F1515 selected window, they remove supervision for 1.38% of tokens while preserving observations and trajectory structure Concept-based masking increases mean reopening time more than sevenfold, from 1.89 to 14.25 seconds, exceeding the mean delay of all comparison methods

Conclusion

The study demonstrates that BSFs recover interpretable, multidimensional features in vision-language-action models, providing tools for steering robot behavior and tracing it to training data Targeted interventions establish these features’ behavioral relevance, while feature-guided data masking selectively delays failure recovery and preserves task performance in their evaluation Robust closed-loop visual and geometric steering will require tracking moving image regions and adapting to changing observations and robot state BSFs are also not uniformly superior, as SAEs perform better on some quantitative metrics, underscoring the importance of unpooled activations and training scale alongside feature geometry

--- Page 5 ---

Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models Aiden Swann⋆,•,a Mozes Jacobs⋆,•,b Mathilde Papillon⋆,•,c Siddharth Boppana• David Chanin• Dron Hazra• Usha Bhalla• Curt Tigges• Mac Schwagera Owen Lewis• Jack Merullo• Ekdeep Singh Lubana • Atticus Geiger • Thomas Fel†,• Matthew Kowal†,•

--- Page 6 ---

the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior

--- Page 7 ---

1 Introduction A vision-language-action (VLA) model maps visual observations, language instructions, and robot state to actions in the physical world (Black et al., 2025; Lee et al., 2026; Fang et al., 2026; Black et al., 2024; Brohan et al., 2013; Kim et al., 2014; Deshpande et al., 13, Deshpande, A. Guru, M. Hendrix, R. Jauhri, S. Eftekhar, A. Tripathi, R. Argus, M. Salvador, J. Fang, H Wallingford, M Kim Y.-C Lee Y.-C Wolters P Rayyan O Zhang M Duan J Farley K Han W Vanderbilt E Fox D Farhadi A Chalvatzaki G Shah D and Krishna R Molmob0t: Large-scale simulation enables zero-shot manipulation. arXiv preprint arXiv:2603.16861, 2026; Lee et al., 2026; Fang et al., 2026; Black et al., 2014; Brohan et al., 2013; Kim et al., 2014 Deshpande A Guru M Hendrix R Jauhri S Eftekhar A Tripathi R Argus M Salvador J Fang H Wallingford M Kim Y.-C Lee Y.-C Wolters P Rayyan O Zhang M Duan J Farley K Han W Vanderbilt E Fox D Farhadi A Chalvatzaki G Shah D and Krishna R Molmob0t: Large-scale simulation enables zero-shot manipulation. arXiv preprint arXiv:2603.16861, 2026; Lee et al., 2026; Fang et al., 2026; Black et al., 2014; Brohan et al., 2013 Deshpande A Guru M Hendrix R Jauhri S Eftekhar A Tripathi R Argus M Salvador J Fang H Wallingford M Kim Y.-C Lee Y.-C Wolters P Rayyan O Zhang M Duan J Farley K Han W Vanderbilt E Fox D Farhadi A Chalvatzaki G Shah D and Krishna R Molmob0t: Large-scale simulation enables zero-shot manipulation. arXiv preprint arXiv:2603.16861, 2026; Lee et al., 2026; Fang et al., 2026; Black et al., 2014; Brohan et al., 2013

--- Page 8 ---

Figure 1: Block-sparse featurizers (BSFs) recover multi-dimensional robotics concepts. Examples from vision-languageaction models encode end-effector position (top), fruit-and-vegetable semantics (middle), and proprioceptive state (bottom). Each point is a token’s 3D feature code, colored by PCA coordinates, task or semantic labels, and sparse-autoencoder (SAE) assignments across columns. Activations from MolmoAct2 L25, L21, L13 (top to bottom). Left overlays share the PCA colors. SAE features tile distinct regions within each block, showing how BSFs group related variation into a single feature

--- Page 9 ---

the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior

--- Page 10 ---

2 From Sparse Directions to Sparse Subspaces The goal of a featurizer is to factorize model activations into a set of interpretable concepts (Huben et al., 2024; Bricken et al., 2023; Gao et al., 2025; Bussmann et al., 2024; Rajamanoharan et al., 2019; Fel et al., 17, Hindupur, S. S. R. Lubana E. Fel T and Ba D E Projecting assumptions: The duality between sparse autoencoders and concept geometry Hindupur et al., 2025

--- Page 11 ---

Block-sparse featurizers The simplest answer is to allow b > 1 A block-sparse featurizer (BSF) (Fel et al., 2026a)

Improvements for AI systems

Bold header: Targeted behavior modification via feature-guided data masking. This system can select features which steer robot behavior to target object on a set discovery inputs and then evaluated, on held-out inputs with fresh noise seeds, allowing for the selective delay of failure recovery by masking tokens where the feature F1515 fires.
Bold header: Compact, interpretable policy steering. The system can perform redirection task in which two objects sit at opposite ends of a table using a minimal set of features, achieving high success rates on held-out data with fewer features than standard SAEs, as demonstrated by the finding that Geometric steering is the most compact: a single BSF block achieves 99.7% recovery.
Bold header: Enhanced understanding of robot internal states through multi-dimensional feature recovery. The AI can recover several multi-dimensional task-relevant features underlying the model’s computation (e.g., perception, language, proprioceptive state, episode phase, and control variables), providing a richer interface for understanding how models process visual and language inputs simultaneously.
Bold header: Data curation informed by causal feature attribution. The system can use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, enabling researchers to target training data with precision, identifying fewer than 1.4% of training tokens whose masking is sufficient to remove the targeted behavior.

Abstract

Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace. We investigate whether recovering this structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn. We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables). We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction. Finally, we use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, identifying fewer than 1.4% of training tokens whose masking is sufficient to remove the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.