Policy-Gradient Transfer under a Shared State Estimator

arXiv:2610.21693 · eess.SY, cs.SY · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "Policy-Gradient Transfer under a Shared State Estimator".

Dev: The gist This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: Moving on to the summary of this paper, we're looking at "Policy-Gradient Transfer under a Shared State Estimator". The main thrust is addressing policy gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator.

Dev: Essentially, they tackle partially observed nonlinear systems where the plant and simulator have different dynamics and observation models, but they share additive Gaussian noise with common known covariance. The controller itself is defined as a squashed Gaussian policy with uniformly bounded latent mean and its first two parameter derivatives.

Taro: So the setup involves comparing the real plant M and the simulator Mc at a fixed policy parameter theta, where their dynamics and observation maps might differ, but they both use a common observer F.

Rosa: The paper claims they derive a bound on the difference between these exact closed-loop policy gradients that's linear in the noise-weighted model error. Crucially, this bound holds without requiring any estimator accuracy guarantee because the shared estimator is just a common post-processing of measurements.

Dev: They rely on the data-processing inequality to compare trajectory laws, which means they can bound the divergence of those laws without any estimation-error term present in their comparison.

Taro: So what this matters for us is that we don't have to worry about needing perfect knowledge of the estimator's accuracy upfront to get a useful estimate on how far off our gradients might be.

Rosa: That’s right. Beyond just bounding the gradient difference, they also derive an occupancy-weighted refinement, which is a sufficient condition for when the simulator gradient is an ascent direction for the plant.

Dev: And they also provide a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run, incorporating optimization and truncation terms and model discrepancies.

Taro: So it gives us both theoretical guarantees on how to use simulation data practically, linking model error to actual gradient transfer performance along visited trajectories.

Rosa: It’s about showing that even with imperfect models and estimators, we can still get a useful bound on the gradient discrepancy if we focus on the noise-weighted error and occupancy.

Dev: So it sets up a framework for making practical decisions about when to trust simulator gradients over physical ones in this kind of setup.

Conclusion: Rosa: So wrapping up with the conclusion for this paper, "Policy-Gradient Transfer under a Shared State Estimator". The authors are Dongming Wang and Wei Ren.

Dev: The title really captures the essence of what they did—policy gradient transfer when both systems share a fixed state estimator. It's about connecting simulation to reality under shared observation constraints.

Taro: So what this means in simpler terms is that we can establish a reliable way to get a gradient direction from the simulator, even if it doesn't perfectly match the real plant dynamics or observation models.

Rosa: In other words, because of their results, we can use simulation data to guide optimization on the physical system with reasonable confidence.

Dev: That confidence isn't based on knowing how good the estimator is exactly; it’s based on controlling how much noise affects the system and focusing our analysis on visited states.

Taro: So it moves us toward a more practical way of using simulators for real-world control, where we focus on what actually matters during training rather than getting bogged down in perfect estimation accuracy metrics.

Rosa: It’s about building trust in the transfer process by showing that the gradient difference is controlled linearly by the noise-weighted model errors.

Dev: And they give us tools—the occupancy-weighted bound and those finite budget bounds—to decide when simulation training is sufficient to guide a physical system effectively.

Taro: It’s a way to manage uncertainty in policy gradient transfer without needing an estimator accuracy guarantee, just by focusing on the noise level and the visited states.

Dongming Wang, Wei Ren

University of California, Riverside

eess.SY, cs.SY

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 74/100

The gist: The gist This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator, deriving bounds on gradient

Key concepts

Policy Gradient Transfer: This is the process of moving a learned control policy from a simulated environment to a real physical system. The challenge here is ensuring that the policy performs well in reality, even though it was trained in simulation. The paper focuses on how this transfer works when both systems rely on the same fixed state estimator.
Shared State Estimator: Both the simulator and the physical plant use a single, identical estimator to process noisy measurements and estimate the system's state. The crucial insight is that because they use the same processing map, trajectory laws can be compared using data-processing inequalities, allowing for gradient analysis without needing perfect knowledge of how accurate that shared estimator is.
Occupancy-Weighted Bound: This refinement uses the plant's actual visited trajectories (occupancy) to weight the model errors. Instead of a worst-case error bound, this method relates the policy gradient discrepancy directly to how often and where the system actually operates in its state space. This provides a more practical bound for assessing performance during training.
Ascent Direction Condition: This is a sufficient condition used to determine if the simulator's gradient points in a direction that improves the plant's performance. The paper derives this by comparing the distance between the two gradients; if this distance is smaller than the norm of the simulator gradient, it suggests that following the simulator's direction will lead to an improvement in actual plant performance.

Summary

The gist This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator, deriving bounds on gradient discrepancy that are linear in noise-weighted model error and require no estimator-accuracy guarantee

Policy Gradient Transfer Under Shared Estimator

The core problem addressed is bounding the difference between the exact closed-loop policy gradients of a plant and a simulator that share a fixed state estimator in terms of forward dynamics and observation model errors, without requiring an estimator-accuracy assumption The authors show that the difference between the exact plant and simulator gradients is at most a constant multiple of the model error, which is defined as the largest noiseweighted discrepancy between the dynamics maps and between the observation maps This bound relies on a key observation that the same Borel map processes the measurements in both environments, so the data-processing inequality bounds the divergence of the trajectory laws without any additional regularity or accuracy assumption on that map.

Key Theoretical Results

The paper derives several crucial results regarding gradient transfer and ascent directions

** First, they show that the difference between the exact plant and simulator gradients is at most a constant multiple of the model error, the largest noiseweighted discrepancy between the dynamics maps and between the observation maps. The key observation is that the same Borel map processes the measurements in both environments, so the data-processing inequality bounds the divergence of the trajectory laws without any additional regularity or accuracy assumption on that map. This establishes a bound on gradient discrepancy with no assumption on a fixed shared estimator beyond measurability. The worst-case model error can be replaced by errors weighted by the plant’s discounted occupancy, which relates gradient transfer to model accuracy along visited trajectories. Furthermore, they show that the simulator gradient is an ascent direction for the plant if its distance from the plant gradient is smaller than its norm. The worst-case error in (7) can be replaced by errors under the plant’s occupancy, leading to a bound in Corollary 1. The final result is that the realized discrepancy depends on the observer and the noise level, which the uniform bound does not resolve.> Page 8> Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and derived an occupancy-weighted bound, a sufficient ascent condition and a finite-budget bound on the expected squared plant gradient at a random iterate of training in the simulator alone Page 7>The paper concludes that they have studied policy-gradient transfer from a simulator to a plant when both are controlled through a shared fixed observer and have shown that the difference between the plant and simulator policy gradients is bounded linearly by the noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee, and

Improvements for AI systems

Improved policy-gradient transfer via Occupancy-Weighted Refinement: This allows deriving a sufficient condition for the simulator gradient to be an ascent direction for the plant, which is a key mechanism for ensuring policy improvement when transferring gradients, as stated in Section III.C.
Occupancy-weighted bound on expected squared plant gradient: The system can utilize Corollary 1 (Occupancy-weighted transfer) to obtain a bound on the expected squared plant gradient at a random iterate of finite simulator-training runs, providing a specific certificate for training performance under Assumptions A1 and A2.
Finite-budget deployment guarantee: Implement Algorithm 1 with the derived bounds to achieve Proposition 2 (Deployment stationarity), which provides an upper bound on the expected squared plant gradient at a random iterate of finite simulator-training runs, contingent on the step size parameter alpha.

Abstract

This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.