Online Inference in Distributional Temporal-Difference Learning

arXiv:2608.14408 · stat.ML, cs.LG · Submitted 2026-08-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Online Inference in Distributional Temporal-Difference Learning".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to wrap up our discussion on "Online Inference in Distributional Temporal-Difference Learning," the authors essentially show that their online multiplier bootstrap consistently reproduces the asymptotic sampling distribution for both smooth and nonsmooth statistical functionals under fixed policy conditions <ref:2608.14408#pg2>.

Jane: That means we can trust our bootstrap inferences for things like variance, CVaR, expected shortfall, and expectiles because they converge to that same Gaussian limit conditionally on the observed trajectory <ref:2608.14408#pg2>.

Lu: The paper provides a framework for inferring return quantiles by developing a local asymptotic theory over T-one/two-neighborhoods of finitely many thresholds, along with its bootstrap analogue <ref:2608.14408#pg3>.

Meng: It's important to remember the caveat they mentioned regarding step sizes: that for nonsmooth functionals, the convergence in P-probability depends on a specific condition on those step sizes <ref:2608.14408#pg2>.

Tom: The paper is about using distributional temporal-difference learning from a single Markov trajectory to estimate these return distributions, and it achieves this through the Polyak–Ruppert averaged estimator <ref:2608.14408#pg2>.

Jane: It really changes how we think about statistical inference in reinforcement learning by showing that online methods can handle these complex functionals in a statistically sound way <ref:2608.14408#pg3>.

Lu: This work is significant because it connects the nonparametric estimation method to concrete inferential results for both smooth and nonsmooth quantities, which is a pretty strong result <ref:2608.14408#pg3>.

Meng: For deployment, it gives us a solid theoretical foundation that we can check against when we're looking at our own real-world traffic data instead of just theoretical benchmarks <ref:2608.14408#pg2>.

Tom: We've covered the main points: the Gaussian limits for smooth cases and the local asymptotic theory for nonsmooth ones, all supported by a robust bootstrap method <ref:2608.14408#pg3>.

Conclusion: Tom: So, we’ve been looking at this paper, "Online Inference in Distributional Temporal-Difference Learning." It’s about figuring out what we are trying to estimate—the return distribution—when we're learning online with a fixed policy.

Jane: Right. So the authors are basically showing how you can get reliable estimates for those distributions even when you only see one path, not a whole history of interactions.

Tom: Exactly. They use something called distributional temporal-difference learning, which is nonparametric, so they aren't assuming any specific shape for the data distribution to start with.

Lu: The core mechanism they focus on is this Polyak–Ruppert averaged estimator, and they prove that when you look at the root- T error of that estimator, it behaves like a centered Gaussian random element in Cramér space.

Meng: A Gaussian limit is good, but what does that mean for me? Because I’m trying to deploy this on real traffic data, what are we actually looking at here? Is this just theory?

Jane: It’s not just theory. They also show that if you use a bootstrap method—the online multiplier bootstrap—you can get the same Gaussian limit, provided you know the observed trajectory.

Tom: And that’s where it gets interesting for inference. They prove this works for smooth functionals, like mean or variance of the return distribution, but they also tackle nonsmooth ones too.

Lu: For those nonsmooth things—like expected shortfall or expectiles—they develop a local asymptotic theory around specific thresholds, and they show that the bootstrap analogue works there too.

Meng: So it means we can actually perform inference on things like CVaR or expected shortfall, not just guess at them based on the data. That’s something practical for production systems.

Jane: It’s about moving from just estimating a distribution to actually being able to make statements about that distribution—like saying this value is likely within this range.

Tom: The main point of the paper is that their multiplier bootstrap consistently matches the sampling distribution for both smooth and nonsmooth cases, which justifies using it for inference on various statistical functionals.

Lu: It’s a solid theoretical foundation connecting online learning to robust statistical estimation under fixed policy constraints.

Yang Peng, Liangyu Zhang

Yau Mathematical Sciences Center, Tsinghua University · School of Statistics and Data Science, Shanghai University of Finance and Economics

stat.ML, cs.LG

Submitted: 2026-08-14

Updated: 2026-10-16

Comments: There is a technical error in "Step 3" of the proof of Theorem 4, section 4

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist: This paper studies online statistical inference for functionals of the return distribution under a fixed policy by proving that its root-T error converges weakly to a centered Gaussian

Key concepts

Distributional Temporal-Difference Learning: This method estimates the return distribution by recursively updating its estimate at each step of an observed Markov trajectory. It uses a stochastic-approximation recursion to learn the distribution without needing a full dataset, making it suitable for online learning scenarios.
Polyak–Ruppert Averaged Estimator: This estimator is used to calculate the functional of interest by applying it to the average of the return distribution estimates obtained during the online learning process. It serves as a key tool for obtaining consistent estimates of statistical functionals.
Cramér Space Central Limit Theorem: This theorem shows that when you normalize the error from an averaged estimator, it converges weakly to a centered Gaussian random element within Cramér space. This provides the precise probabilistic limit for smooth statistical functionals.
Local Asymptotic Theory for Nonsmooth Functionals: For complex, nonsmooth functionals like expected shortfall, this theory establishes an asymptotic representation around specific thresholds. It allows researchers to perform inference on these quantities by examining local expansions of the estimated return Cumulative Distribution Function.

Summary

The gist: This paper studies online statistical inference for functionals of the return distribution under a fixed policy by proving that its root-T error converges weakly to a centered Gaussian random element in Cramér space and that bootstrap inference is valid for smooth and nonsmooth statistical functionals.

Distributional Temporal-Difference Learning

The paper investigates online statistical inference for functionals of the return distribution estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory The estimator recursively updates the return-distribution estimate at each step of the observed trajectory using a stochastic-approximation recursion The Polyak–Ruppert average is used to estimate the functional by applying it to this average

Inference for Smooth Functionals

For smooth functionals, the analysis establishes an asymptotic linear representation and a Gaussian limit for the averaged estimator The Cramér-space central limit theorem shows that the normalized error converges to a centered Gaussian random element in Cramér space with covariance operator A−1Σ(A−1)∗ The online multiplier bootstrap is shown to consistently reproduce this Gaussian limit conditionally on the observed trajectory

Inference for Nonsmooth Functionals

For nonsmooth functionals characterized by CDF equations, the paper develops a local asymptotic theory over T−1/2-neighborhoods of finitely many thresholds This theory allows inference for return quantiles through local CDF expansions The bootstrap analogue is also developed to ensure its validity in this nonsmooth setting

Bootstrap Validity

The online multiplier bootstrap is shown to reproduce the same Gaussian limit conditionally on the observed trajectory for smooth functionals The conditional law of the normalized bootstrap error converges to the same Gaussian limit as the sampling distribution For nonsmooth functionals, Theorem 5 shows that this convergence holds in P-probability under a specific condition on step sizes

Inferential Consequences and Examples

The paper provides examples of common smooth statistical functionals such as the mean and variance of the discounted return distribution It also details how to handle nonsmooth functionals like expected shortfall and expectiles using local CDF expansions Theorem 6 establishes that these CDF-defined functionals can be estimated by relating the root-T localization of the estimators to Gaussian limits.

Conclusion

The paper concludes that the online multiplier bootstrap consistently reproduces the asymptotic sampling distribution of the Polyak–Ruppert averaged estimator for both smooth and nonsmooth statistical functionals under fixed policy conditions The results justify bootstrap inference for various statistical functionals, including variance, CVaR, expected shortfall, and expectiles. It also provides a framework for inferring return quantiles through local asymptotic theory. The paper demonstrates that the conditional weak convergence holds in P-probability for both sampling and bootstrap procedures under appropriate conditions.

--- Page 60 ---

[1] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, pages 449–458, 2017

[2] M. G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning. MIT Press, 2023

[3] M. B¨ock and C. Heitzinger. Speedy categorical distributional reinforcement learning and complexity analysis. SIAM Journal on Mathematics of Data Science, 4(2):675–693, 2022 doi:10.1137/20M1364436

[4] Y. Chandak, S. Niekum, B. da Silva, E. Learned-Miller, E. Brunskill, and P. S. Thomas. Universal off-policy evaluation in Advances in Neural Information Processing Systems volume 34 pages 27475–27490 2021

[5] A. Huang, L. Leqi, Z. C. Lipton, and K. Azizzadenesheli. Off-policy risk assessment for markov decision processes in Proceedings of the 25th International Conference on Artificial Intelligence and Statistics volume 151 pages 5022–5050 2022

[6] K. Jin, Y. Peng, J. Yang, and Z. Zhang. Accelerated distributional temporal difference learning with linear function approximation arXiv preprint arXiv:2511.12688 2025

[7] E. C. Kaya and A. Hashemi. A finite-iteration theory for asynchronous categorical distributional temporal-difference learning arXiv preprint arXiv:2605.06866 2026

[8] T. Morimura, M. Sugiyama, H. Kashima, H. Hachiya, and T. Tanaka. Nonparametric return distribution approximation for reinforcement learning in Proceedings of the 27th International Conference on Machine Learning pages 799–806 2010

[9] Y. Peng, L. Zhang, and Z. Zhang. Statistical efficiency of distributional temporal difference learning in Advances in Neural Information Processing Systems volume 37 pages 24724–24761 2024 doi:10.52202/079017-0779

[10] Y. Peng, K. Jin, L. Zhang, and Z. Zhang.

Improvements for AI systems

Distributional policy evaluation for smooth functionals can be performed by showing that the functional delta method transfers the Gaussian and bootstrap limits to smooth functionals (Page 13), enabling inference for quantities like variance, CVaR, expected shortfall, and expectiles.
Inference for nonsmooth statistical functionals is enabled by developing a local asymptotic theory for the estimated return CDF over T − 1/2-neighborhoods of finitely many thresholds, which allows for the estimation of return quantiles.
The online multiplier bootstrap consistently reproduces the asymptotic sampling distribution conditionally, as shown by Theorem 2: the conditional law of Z∗T converges to the same Gaussian law (Page 11). This justifies using bootstrap inference for these functionals without resampling.
For quantile estimation, the paper provides a specific result: The generalized-inverse estimators qˆT,τ,s and qˆ∗ T,τ,s are Op(1) in probability (Page 57), allowing for robust estimation of return quantiles even when the exact CDF is not attained.
Policy evaluation systems can be designed to provide confidence intervals with P[Φ(ηπ) ∈ I∗T,1−α] → 1 − α, where the interval is constructed from the basic bootstrap confidence interval (Page 12).

Abstract

We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root- T error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root- T difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over T-1/2-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.