Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation

arXiv:2606.06281 · cs.RO · Submitted 2026-06-04 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation".

Dev: Here's the gist of the paper. The gist The MiTaS framework presents a novel representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks, achieving an average success rate of 80% How it works MiTaS is a comprehensive representation framework designed to condition a flow matching imitation learning policy to leverage multiple tactile sensors operating at different temporal resolutions to solve complex manipulation tasks.

Rosa: The key insight of the method is that different tactile modalities capture complementary aspects of contact-rich interactions, where standard vision-based sensors provide high spatial resolution of contact geometry and event-based sensors capture high-frequency temporal dynamics The architecture involves three main components: 1.

Dev: a novel architecture utilizing modality-specific CNN stems that encode diverse sensor inputs into unified token representations.

Title and authors: Rosa: So we've covered what MiTaS is and how they set up their architecture, and now let's look at the actual summary of the Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation paper to get a clearer picture of what they achieved.

Dev: The core summary is about using this novel representation framework to condition a flow matching imitation learning policy that leverages multiple tactile sensors operating at different temporal resolutions, which they argue solves complex contact-rich manipulation tasks with an average success rate of eighty percent.

Taro: So the main point is that by looking at both the high spatial detail from vision and the high frequency dynamics from event-based sensors, they create a representation that captures complementary aspects of contact interactions.

Rosa: That's right; vision gives you the geometry, and Evetac gives you those fast contact changes like incipient slip or vibration that frame-based sensors miss because they smooth them out.

Dev: The architecture uses modality-specific CNN stems to turn these different sensor inputs into unified tokens, which are then fused through a transformer encoder with cross-attention for rich multisensory features.

Taro: It sounds like they're essentially building a sophisticated sensory pipeline that translates heterogeneous raw data into something the policy can actually use for control.

Rosa: Precisely. They also have this multitactile co-training scheme where they train the visual and GelSight policy while exploiting the Evetac sensor exclusively during training to improve performance when only a subset of sensors is available at inference time.

Dev: The implication there is that you can train a robust visual-tactile policy that benefits from cross-modal feature alignment established during training, even when you have to rely on fewer sensors in deployment.

Taro: It’s about learning how to use the high-frequency data as a regularizer for the lower resolution features, which is a really clever way to make the system more resilient.

Rosa: So they're proposing this as a comprehensive representation framework that conditions their flow matching policy for action generation, giving them this rich input to predict future robot commands.

Dev: And the training objective involves optimizing the network parameters so that its output velocity field accurately regresses a straight-line probability path and its corresponding constant target velocity field.

Taro: So they're using generative modeling, specifically flow matching, to guide the policy into generating smooth, predictable movements based on the sensory inputs.

Rosa: It's a lot of moving parts: multi-resolution encoding, transformer fusion, flow matching action generation, and co-training for robustness during training.

Dev: And when you look at the experimental evaluation across five tasks—Gear assembly, Board Wiping, Lamp Installation, Key in Lock and Lightbulb Connection—the results show MiTaS outperforms a strong multi-sensory baseline and two vision-only approaches in every task.

Taro: So the key result is that MiTaS actually works where vision alone just fails because of occlusion or complex contact dynamics, achieving that eighty percent success rate on average.

Rosa: That means tactile sensing isn't just nice to have; it’s necessary for these high-precision manipulation scenarios.

Dev: And the sensor ablation study confirmed that visual multi-tactile models achieve the highest success rate regardless of model design, which is a strong finding for this paper.

The paper's summary: Taro: So what are the specific improvements they suggest in this Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation paper beyond just presenting their initial setup?

Rosa: One major improvement is using the multi-resolution encoding architecture where they use modality-specific CNN stems to extract sensor-specific representations before fusing them with a transformer encoder.

Dev: This way, instead of just throwing raw data into a single fusion network, they create these distinct tokens that are then augmented with positional and modality embeddings for better spatial and source awareness.

Taro: And the augmentation part, where they define z'i as z i plus a learned position vector p g(i) and a modality-specific embedding e s(i), seems like it helps the fusion network understand exactly where a token came from.

Rosa: That's right, it separates where the token lies in the spatial layout from which sensor produced it, which feeds into the main transformer encoder that does all this cross-attention and MLP layers.

Dev: Then they also have the flow matching policy itself being defined by a Diffusion-Transformer or DiT head that regresses a time-conditioned velocity field along a straight noise-to-action path, which is more structured than just standard regression.

Taro: So this structural definition of the action generation using flow matching means the network is learning to generate the velocity field in a way that follows a learned probability path, not just any random output.

Rosa: And finally, the multi-tactile co-training scheme itself is an improvement because it allows them to train a visual-tactile policy while exploiting the high-frequency Evetac sensor during training.

Dev: That co-training scheme specifically improves performance by regularizing standard visual features to latently encode and compensate for missing features from the Evetac observations during inference.

Taro: So it’s not just about adding sensors; it's about designing a training procedure that actively trains the system to be resilient when data is missing, which makes sense for real-world deployment.

Rosa: In short, they improved things by creating this structured way of encoding inputs and training the policy with this specific co-training strategy to ensure better generalization.

The paper's improvements: Dev: So to wrap up on the Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation paper, we've seen how they used a novel representation framework to combine multi-resolution tactile sensors and flow matching policy conditioning.

Rosa: They managed this by using modality-specific CNN stems for encoding and then fusing those tokens through a transformer encoder with cross-attention to create these rich multisensory features that condition the policy.

Taro: It’s about building a system that can handle the physical complexity of contact interactions by blending high spatial detail from vision with the speed of event-based sensors.

Dev: The results show they got an average success rate of eighty percent across five challenging tasks, which is a solid number compared to vision-only baselines that struggled on certain scenarios.

Rosa: And the implication for us is that tactile sensing is a necessary tool for these complex manipulation jobs, and MiTaS provides a way to condition policies directly from these fused features.

Taro: For my perspective, it means future robots will have to learn not just how to follow instructions but also how to feel and adapt their movements in real-time when the environment misbehaves.

Dev: That's the takeaway; we need systems that are robust enough for things that aren't perfectly modeled.

Rosa: We’re moving toward policies where they can use this fused sensory information to predict delta actions directly from these multimodal sensory inputs without needing explicit robot state conditioning, which is a pretty neat feature.

Taro: I just think the challenge moving forward is making sure those features generalize across different types of physical interactions beyond the five tasks they tested.

Dev: Yeah, that’s where we need to see how this architecture handles things when the world throws us something completely new in front of it.

Conclusion: Rosa: So MiTaS is tackling complex contact manipulation tasks by using multiple tactile sensors operating at different temporal resolutions—vision for geometry and event-based sensors for fast dynamics—to condition a flow matching imitation learning policy.

Dev: The core idea is fusing those inputs through a transformer encoder using modality-specific CNN stems to get unified tokens that the policy then conditions on.

Taro: And they use a multi-tactile co-training scheme where they train the visual and GelSight parts while exploiting the high frequency Evetac sensor during training to regularize the system for inference.

Rosa: The results show an average success rate of eighty percent across five different tasks, which is pretty solid when you compare it to vision-only models that just fail on tricky contact scenarios.

Dev: I'm interested in how they handle the real-time loop and latency there; if we have to run this on a low-power robot, what’s the safety case for that policy generation speed?

Taro: When things go wrong in those contact rich tasks, MiTaS seems pretty robust because of that co-training. It lets the model learn how to compensate for missing data.

Rosa: That makes sense; if you only have GelSight and vision at inference time, the policy should still perform reasonably well because it learned from seeing the Evetac data during training.

Dev: Still, they flag something important: when you need precise object in hand location information, even with that co-training, performance can dip on tasks like key in lock.

Taro: Yeah; if the robot needs to know exactly where it's holding something for fine alignment, relying only on those sensors might not be enough.

Rosa: So MiTaS shows a strong capability for general contact manipulation, but maybe it’s not perfect for every single physical situation yet.

Dev: It definitely points toward needing that high temporal resolution data to get the full picture of what’s happening during a delicate interaction.

Taro: Anyway, MiTaS is a good step forward in using heterogeneous sensors for imitation learning; next up, we're looking at how other papers are handling the readiness barrier functions in multirotor allocation.

Rickmer Krohn, Erik Helmut, Niklas Funk, Jan Peters, Vignesh Prasad, Georgia Chalvatzaki

Interactive Robot Perception & Learning at TU Darmstadt · Intelligent Autonomous Systems at TU Darmstadt

cs.RO

Submitted: 2026-06-04

Updated: 2026-10-21

Comments: 20 pages, 5 figures, CoRL 2026

Project page: http://mitas-touch.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The MiTaS framework presents a novel representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich

Key concepts

Multi-Resolution Tactile Encoding: This component uses separate CNN stems for each sensor type (vision, GelSight, Evetac) to process their unique data formats. Each stem creates tokens representing the sensor's specific input, which are then combined using a transformer with cross-attention. This ensures that spatial details from vision and temporal dynamics from event sensors are both captured and unified into a single set of features for the policy.
Flow Matching Policy: This is the method used to generate robot actions. Instead of traditional imitation learning, MiTaS uses a flow-matching generative model. It learns to predict a velocity field that smoothly transitions from random noise to the desired action path. This allows the policy to generate precise delta commands for movement directly from the fused sensory tokens.
Multi-Tactile Co-Training: This is a training strategy where the model is trained using both high-frequency Evetac data and lower-frequency visual/GelSight data simultaneously. During training, it learns to compensate for missing high-speed data by relying on the other sensors. This results in a streamlined policy that performs well even when the fastest sensor (Evetac) is removed during actual robot use.

Terminology

Downstream Tasks — 1× in this paper · explained in 2 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The MiTaS framework presents a novel representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks, achieving an average success rate of 80%

How it works

MiTaS is a comprehensive representation framework designed to condition a flow matching imitation learning policy to leverage multiple tactile sensors operating at different temporal resolutions to solve complex manipulation tasks. The key insight of the method is that different tactile modalities capture complementary aspects of contact-rich interactions, where standard vision-based sensors provide high spatial resolution of contact geometry and event-based sensors capture high-frequency temporal dynamics

The architecture involves three main components:

a novel architecture utilizing modality-specific CNN stems that encode diverse sensor inputs into unified token representations.
a flow-matching policy for action generation and visuomotor control conditioned on these tokens.
a multitactile co-training scheme that leverages all available sensors during training to improve downstream policy performance when only a subset of sensors is available at inference time.

Multi-Resolution Tactile Encoding Architecture

The encoder extracts and fuses features from heterogeneous sensor modalities using a transformer-based architecture with cross-attention to condition the policy. This encoder consists of sensor stems, positional / modality encoding, and sensor fusion.

The Sensor Stems process each modality by tokenizing it with a dedicated CNN-stem to extract a meaningful sensor-specific representation. For the wrist-camera, this involves stacking two RGB frames along the channel axis and applying four strided 2D convolutions, producing a 16×16 grid of D=256-dimensional tokens. The GelSight stem mirrors this design at a rectangular 120 × 160 resolution and yields a 12 × 16 token grid. For Evetac’s 16-frame stack, a four-layer 3D-convolutional stem jointly compresses time and space into a 12 × 16 grid, reflecting the higher temporal density of that modality.

Positional and modality encoding augment each sensor stem output by adding a learned position vector pg(i) that encodes its location within that sensor’s token grid, and a modality-specific embedding es(i) where s(i) ∈ 3 identifies the source sensor. The augmented token is then defined as z′ i = zi + pg(i) + es(i), which separates where a token lies in the spatial layout from which modality produced it.

Sensor Fusion is achieved by processing the augmented visual multi-tactile token sequence through a transformer encoder with multi-head self-attention, MLP-layers, pre-normalization, and residual connections. This fusion transformer allows for full self- and cross-attention of sensors, leading to rich multisensory features forming the conditioning for the policy.

Action Generation through Flow Matching

The visual multi-tactile embeddings are utilized for robot action generation by leveraging a conditional flow matching generative model. The policy output is defined as a fixed-length sequence of future robot commands At = [at, at+1,..., at+H−1] ∈ R H×da.

The individual actions are delta commands in the robot local frame, i.e., a = ∆p = [∆x, ∆y, ∆z, ∆ψ] ∈ R 4. This formulation controls translation and the yaw rotation around the z-axis while keeping pitch and roll fixed. The policy head is a Diffusion-Transformer (DiT) that regresses a time-conditioned velocity field along a straight noise-to-action path.

Flow-matching training and inference involve defining a straight-line probability path xt and its corresponding constant target velocity field ut for t ∈ [0, 1] as xt = t · x1 + (1−t) · x0, and ut = dxt/dt. The training objective is to optimize the network parameters θ such that its output vˆθ(xt, t, C) accurately regresses this target velocity field. During inference, actions are generated by integrating the learned velocity field via explicit Euler integration starting from an initial noise vector, x0 ∼ N (0, I).

The DiT action head outputs vˆθ(xt, t, C), which operates on an intermediate action state xt ∈ R H×da. Each horizon slot is linearly projected to a D-dimensional token and summed with a learned positional embedding over the chunk, yielding Xt ∈ R H×D. A stack of L DiT blocks updates Xt while conditioning on multisensory tokens C ∈ R Ncond×D produced by the perception encoder.

Multi-Tactile Co-Training

The proposed scheme is an asymmetric co-training procedure designed to train a visual-tactile policy (utilizing vision and GelSight) while exploiting the high-frequency Evetac sensor exclusively during the training phase. Each training batch is constructed by sampling half of the sequences with Evetac observations and the remaining half without.

This stochastic co-training procedure allows the high-frequency temporal dynamics captured by Evetac to shape the representation space of the sensor-fusion network. Because the model must simultaneously learn to generate accurate actions in the absence of Evetac, this procedure effectively regularizes standard visual-tactile features to latently encode and compensate for missing features extracted from Evetac observations. The Evetac encoder is omitted entirely during deployment, leaving a streamlined (co-trained) vision-and-gelsight policy that benefits from cross-modal feature alignment established during the training phase.

Experimental Evaluation

MiTaS was evaluated on five challenging contact-rich manipulation tasks: Gear assembly, Board Wiping, Lamp Installation, Key in Lock, and Lightbulb Connection. The experiments investigated overall performance compared to a strong multi-sensory baseline and vision-only approaches

The results showed that MiTaS outperforms the multimodal baseline Sparsh-X and two vision-only ViT-baselines in every task. MiTaS achieves an average success rate of 80%, whereas vision-only models failed, due to occlusion and complex contact dynamics, highlighting the need for tactile sensing in contact-rich manipulation.

The sensor ablation study confirmed that visual multi-tactile models achieve the highest success rate regardless of model design. While simple vision-tactile policies are able to solve most of the task over half of the time, Evetac as the sole tactile sensor provides useful information in certain task, but fails completely when object in hand location is needed. Gelsight, on the other hand, can align the gear and key from various initial states.

The multi-tactile co-training scheme boosted MiTaS performance in 3/5 tasks, while being helpful in 2/5 tasks for Sparsh-X. In tasks like Key in Lock, the co-training scheme lowers performance as it disturbs Gelsight’s features for in-hand localization.

Attention Analysis with Sensor Visualization

Attention analysis reveals how MiTaS uses the different sensors over the course of a task. For instance, in Lamp Installation, the policy attends on vision during reaching, on Gelsight for the prolonged screwing motion, and on all sensors during insertion.

Improvements for AI systems

Improved policy performance in contact-rich tasks by leveraging multi-resolution tactile features, achieving an average success rate of 80% compared to vision-only baselines that fail reliably on certain tasks.
Enhanced ability to handle fast contact changes like incipient slip and high-frequency vibration due to the integration of a high temporal resolution event-based sensor (Evetac) alongside a frame-based sensor (GelSight Mini).
Development of a multi-tactile co-training scheme that boosts visual-tactile policy performance by over 10% in certain tasks, allowing the model to compensate for missing high-frequency data during policy evaluation.
Creation of an encoder architecture using modality specific CNN stems and transformer based fusion to extract and fuse heterogeneous tactile features from RGB camera streams, GelSight Mini sensors, and Evetac sensors into unified tokens.
Implementation of a flow matching policy conditioned on these multi-sensory tokens to predict delta actions directly from high dimensional multimodal sensory input without needing explicit robot state conditioning.

Abstract

Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose an effective architecture using modality-specific convolutional stems and transformer-based fusion that effectively integrates information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multisensory representation then conditions a flow-matching policy for solving downstream tasks. Our systematic study across five contact-rich manipulation tasks demonstrates the effectiveness of multi-resolution tactile features in imitation learning wherein MiTaS achieves an average success rate of 80 %, while vision-only (35.5,%) and visuotactile (55.5,%, 63,%) baselines cannot solve the task reliably. Project Page: http://mitas-touch.github.io.