Sampling-Based Safe Reinforcement Learning

arXiv:2605.19469 · cs.LG, cs.AI, cs.RO · Submitted 2026-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Sampling-Based Safe Reinforcement Learning".

Jane: The gist: Sampling-Based Safe Reinforcement Learning (SBSRL) is a model-based RL algorithm that maintains safety throughout learning by enforcing constraints jointly across a finite set of dynamics samples,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: Now, let's look at the title and who came up with this work, "Sampling-Based Safe Reinforcement Learning."

Tom: It sounds very descriptive of what the paper is trying to achieve—sampling based safety—and it points toward a model-based approach that uses samples to keep things safe.

Lu: The authors are quite a team of experts from places like ETH Zurich, and they’re tackling the fundamental challenge that safety remains a major hurdle for deploying reinforcement learning agents in the real world.

Meng: When you look at the authors, it tells you this isn't just someone playing around with RL; it’s a solid group working on applying advanced mathematics to solve concrete deployment problems.

Lalam: The paper itself suggests that safety exploration is a fundamental challenge because standard methods often struggle to maintain safety while also effectively exploring new things.

The paper's summary: Tom: So, what the paper actually does is propose this model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints across a finite set of dynamics samples.

Jane: It’s about approximating an intractable worst-case optimization over uncertain dynamics by instead enforcing those safety constraints on each sample individually.

Lu: They use a set of M dynamics samples, denoted as f1n through fMn, and they enforce tightened safety constraints jointly across the rollouts generated by all of them.

Meng: This means they aren't solving one monster optimization; they are managing a collection of smaller ones that together approximate the big problem.

Tom: And to handle exploration, instead of adding an explicit bonus to the objective, they encode exploration as a constraint that forces the agent to collect enough information along its trajectories.

Jane: This constraint kicks in only when the greedy policy isn't providing enough information, which is a smarter way to trigger exploration than just adding some extra reward signal.

Lalam: So, instead of trying to force the AI to explore more by giving it a bigger reward, they make it explore because it needs more data to be confident and safe.

The paper's improvements: Tom: The authors suggest a few key improvements over what was already out there. One big one is moving away from trying to solve the worst-case problem explicitly and instead enforcing safety on each sample of the dynamics.

Jane: They are also tackling exploration by using epistemic uncertainty as a constraint, which makes the exploration strategy simpler and more robust for constrained settings.

Lu: They bridge a gap with existing work because they apply this principle—to the best of our knowledge, for the first time—to continuous-domain CMDPs, solving them jointly across all sampled dynamics.

Meng: This joint solution is key because other CMDP methods often use heuristics to approximate the worst-case problem, which usually sacrifices the original theoretical guarantees they started with.

Tom: And finally, they establish theoretical guarantees for online learning, proving high-probability safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy.

Jane: That finite time bound is pretty significant because it shows that we can actually get a near-optimal policy back in a limited number of steps under the right conditions.

Lalam: The paper’s ability to provide these theoretical guarantees, especially when using Gaussian process dynamics, gives us a strong foundation to build more trustworthy and reliable AI systems.

Conclusion: Tom: So, to wrap up the paper "Sampling-Based Safe Reinforcement Learning," it combines sampling-based constraint satisfaction with an exploration constraint on epistemic uncertainty.

Jane: It’s a novel algorithm that ensures safety by enforcing constraints over a finite set of sampled dynamics models and provides a simple regret bound for encouraging exploration through uncertainty constraints.

Lu: The implication is that we now have a way to handle the intractability of worst-case optimization in CMDPs by leveraging the properties of Gaussian process priors to get high-probability safety guarantees.

Meng: For practical engineering, this means we have a scalable neural ensemble variant that allows us to apply this method to high-dimensional continuous control problems on real robotic hardware.

Lalam: This work moves us toward designing AI systems that are inherently cautious, where exploration is driven by the need for information rather than just chasing bigger rewards.

ETH Zurich

cs.LG, cs.AI, cs.RO

Submitted: 2026-05-19

Updated: 2026-10-09

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist: Sampling-Based Safe Reinforcement Learning (SBSRL) is a model-based RL algorithm that maintains safety throughout learning by enforcing constraints jointly across a finite set of dynamics

Key concepts

Constrained Markov Decision Processes (CMDPs)
CMDPs are decision problems where the goal is to maximize expected reward while ensuring an expected cost stays below a specified threshold. This framework is used here to define the problem of finding a safe policy that balances high performance with low risk, which is essential for real-world applications.
Safety Enforcement via Sampling
Instead of checking every possible future scenario, SBSRL enforces safety by testing the policy against a finite set (M) of sampled dynamics models. By tightening the safety constraint across these samples, it approximates the difficult worst-case optimization over all possible uncertain dynamics.
Exploration as a Constraint
SBSRL encourages exploration not through extra rewards, but by requiring the agent to satisfy a minimum epistemic uncertainty threshold at each step. This means exploration is only prioritized when the current policy is not yet sufficiently informative, providing a principled stopping condition for learning.

Terminology

Summary

The gist: Sampling-Based Safe Reinforcement Learning (SBSRL) is a model-based RL algorithm that maintains safety throughout learning by enforcing constraints jointly across a finite set of dynamics samples, which approximates an intractable worst-case optimization over uncertain dynamics

How it works

SBSRL is a model-based RL algorithm that learns an uncertainty-aware dynamics model, leveraging its epistemic uncertainty to drive exploration while also maintaining safety at all times It enforces safety on each sample of the dynamics rather than solving the worst-case problem explicitly, by maintaining a set of M dynamics samples f1n,..., fM n and enforcing tightened safety constraints jointly across their corresponding model-generated rollouts Rather than adding an exploration bonus to the objective, SBSRL encodes exploration as a constraint that requires the agent to collect sufficient information along its trajectories

Key components and assumptions

The framework is built upon Constrained Markov Decision Processes (CMDPs) defined by a tuple ⟨X, A, T, p, r, c, d, ρ0⟩ where the goal is to find a policy that maximizes the expected reward return subject to the expected cost remaining below a threshold d The algorithm relies on several assumptions for theoretical guarantees These include Gaussian noise, access to an RKHS with bounded norm for prior models, and a feasible safe initialization.

Safety enforcement via sampling

The core mechanism involves approximating the worst-case optimization problem by enforcing safety on each sample of the dynamics The safety constraint is enforced by solving a worst-case optimization given by max f∈Fn Jc(π, f) ≤ d, where Fn:= F n:=

max f in F n Jc(π, f) ≤ d This is approximated over the class Fn with a finite set of M sampled models ˜f1,..., ˜fM drawn i.i.d. from the GP prior Truncating unlikely samples f m n to lie within the intersection of all previous GP confidence sets ensures that at least one sample is ζ-close to the true dynamics f⋆ with high probability The constraint tightening Jc(π, fm n) ≤ d - ∆ζ ∀m = 1,..., M guarantees safety under the true dynamics with high probability

Exploration as a constraint

SBSRL promotes exploration via a constraint on the epistemic uncertainty rather than explicit exploration bonuses Specifically, at episode n, the agent requires the policy to satisfy Jsn(πn, µn) ≥ d nσ, where d nσ is a prescribed exploration threshold defined in Theorem 2 This constraint becomes inactive when the greedy policy is already sufficiently informative Infeasibility of this constraint provides a natural stopping criterion, certifying that remaining policies induce uniformly low epistemic uncertainty and linking the exploration threshold directly to the final suboptimality guarantee

Theoretical guarantees

The paper proves that with high probability, SBSRL satisfies the constraint at all episodes and terminates after a finite number of steps Theorem 1 (Safety) shows that for any policy π ∈ Πn safe, Jc(π, f⋆) ≤ d Theorem 2 (Optimality) establishes that there exists a problem-dependent constant C such that the algorithm terminates at iteration n⋆ where Jsn⋆(π, µn⋆) < dn⋆σ ∀π ∈ Πn⋆ safe Furthermore, it guarantees near-optimality upon termination, showing that the policy πˆ returned by Algorithm 1 satisfies Jr(˜π⋆, f⋆) − Jr(ˆπ, f⋆) ≤ ε

Empirical validation

SBSRL is empirically validated both in simulation and on hardware Experiments show that SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware The deep RL variant can be scaled to high-dimensional continuous control tasks, as demonstrated on a highly-dynamic remote-controlled race car operating at 60 Hz SBSRL maintains safety throughout training while improving rewards over the safe baselines in various environments like PENDULUMSWINGUP and CARTPOLESWINGUP The results show that SBSRL achieves near-optimal performance while satisfying the safety constraint at every iteration n

Conclusion

SBSRL is a novel model-based RL algorithm that combines sampling-based constraint satisfaction with an exploration constraint It ensures safety throughout learning by enforcing constraints over a finite set of sampled dynamics models When the dynamics are represented by Gaussian processes, this yields high-probability safety guarantees Furthermore, it provides a simple regret bound that leverages a novel strategy for encouraging exploration in CMDPs, namely a constraint on the epistemic uncertainty which encourages exploration only when the greedy policy is insufficiently informative It is shown to be scalable to high-dimensional continuous control tasks through deep ensemble implementations Finally, this work provides a theoretically sound yet practical algorithm for safe exploration in CMDPs.

Improvements for AI systems

  1. textbfEnforcement of safety through joint constraint satisfaction across dynamics samples: SBSRL enforces safety using a finite number of dynamics samples, and promotes exploration via a constraint on the epistemic uncertainty. This allows for enforcing safety on each sample of the dynamics by maintaining a set of M dynamics samples and enforcing tightened safety constraints jointly across their corresponding model-generated rollouts.

  2. textbfConstraint-driven, uncertainty-based exploration strategy: SBSRL leverages its epistemic uncertainty to drive exploration while also encoding exploration as a constraint that requires the agent to collect sufficient information along its trajectories. This enables exploration only when the greedy policy is not informative enough, eliminating the need for explicit exploration bonuses.

  3. textbf. Near-optimal policy recovery with finite sample complexity: Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. This provides a mechanism to find policies that are near-optimal upon termination, as shown by Theorem 2, which proves that the returned policy satisfies max π∈Π⋆,cξ Jr(π, f⋆) − Jr(ˆπ, f⋆) ≤ ε in finite time.

  4. textbf. Robustness to model misspecification via Gaussian Process dynamics: When the dynamics are modeled using a Gaussian process, SBSRL guarantees safety with high probability. This leverages GP regression to maintain well-calibrated models where the true dynamics lie within the confidence intervals induced by the posterior mean and variance with high probability, which is then used to approximate worst-case optimization.

  5. textbf. Scalability and hardware deployment via deep ensembles: We further introduce a scalable neural ensemble variant that enforces constraints jointly across multiple dynamics and evaluate it in SafetyGym and RWRL [11, 12]. This allows the algorithm to scale to high-dimensional continuous control problems by using a probabilistic ensemble of neural networks where safety is enforced by conditioning the cost critic on the index of the sample.

Abstract

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Sources

Related papers