Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL
cs.LG
Submitted: 2026-08-31
Updated: 2026-08-31
Code: https://github.com/M-Korniak/action-chunked-contrastive-rl
License: http://creativecommons.org/licenses/by/4.0/
The gist: While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions
Terminology
Abstract
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.
Sources
- TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
- Learning to Understand Goal Specifications by Modelling Reward
- Distributed Distributional Deterministic Policy Gradients
- Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
- Training-Time Action Conditioning for Efficient Real-Time Chunking
- Learning Successor States and Goal-Dependent Values: A Mathematical Viewpoint
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- Metrics for Finite Markov Decision Processes
- Grounded Language Learning in a Simulated 3D World
- Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning
- Decoupled Q-Chunking
- Reinforcement Learning with Action Chunking
- Flow Matching for Generative Modeling
- Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making
- Representation Learning with Contrastive Predictive Coding
- Foundation Policies with Hilbert Representations
- Adaptive Action Chunking via Multi-Chunk Q Value Estimation
- State Representation Learning for Goal-Conditioned Reinforcement Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks