Training Hybrid Quantum Policies with Classical Surrogates for Actor-Critic Reinforcement Learning

arXiv:2503.09119 · quant-ph · Submitted 2025-03-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Quantum Radio. Generated commentary on the latest quantum physics and condensed matter papers.

Kai: We're looking at the paper: "Training Hybrid Quantum Policies with Classical Surrogates for Actor-Critic Reinforcement Learning".

Mira: Here's the gist of the paper. The gist The method introduces qtDNN, a tangential surrogate that locally approximates a PQC with a small differentiable network trained on-the-fly from the same minibatch, which enables scalable batch gradients while keeping the original quantum layer for inference The core problem addressed is the prohibitive cost of backpropagating through PQCs to train surrounding classical DNNs This bottleneck makes hybrid training impractical for realistic RL workloads because every gradient step requires a prohibitive number of circuit evaluations and thus erodes the potential quantum advantage Existing gradient estimation techniques make such hybrid training impractical for realistic RL workloads, because every gradient step requires a prohibitive number of circuit evaluations and thus erodes the potential quantum advantage The authors introduce hDQNN-TD3, a hybrid deep quantum neural network for continuous-control reinforcement learning based on the TD3 architecture, which matches or exceeds state-of-the-art classical performance on popular benchmarks This method opens a path toward applying hybrid quantum models to large-scale RL and other gradient-intensive machine learning tasks The goal is to provide a mini-batch-local differentiable approximation of the PQC used only in the backward pass, while the forward path and inference remain quantum This enables scalable mini-batch training by supplying gradients without additional quantum evaluations, compatible with high-throughput GPU pipelines The method offers theoretical support through a local gradient-fidelity guarantee, ensuring that surrogate back-propagation perturbs upstream gradients only within controlled tolerances The empirical effectiveness is demonstrated on the Humanoid-v4 benchmark, outperforming widely adopted classical baselines (Zhao, 2024; 2023a;b) and validating the nontrivial modeling capacity of PQCs compared with fully connected, random, or zero-layer alternatives The hDQNN-TD3 actor matches or exceeds state-of-the-art classical performance on popular benchmarks The qtDNN surrogate provides gradients of sufficient fidelity throughout training, while the late-stage divergence illustrates the head-room the quantum-enhanced actor can exploit once it enters the high-performance regime The results show that in the best case, a single well-tuned hDQNN-TD3 seed can match the ceiling performance of the strongest classical baselines while requiring fewer interaction steps to get there The qtDNN surrogate supplies gradients of sufficient fidelity to stabilize late-stage policy refinement The qtDNN surrogate is trained on-the-fly from the same minibatch, which accelerates qtDNN’s convergence to a moving approximation to its target quantum layer (cf.

Kai: Haarnoja et al.

Mira: (2018)) The qtDNN learns on-the-fly to emulate the quantum circuit while simultaneously providing the actor with differentiable expectation values This allows batched back-propagation through the classical layers without repeatedly querying the quantum circuit for gradients The qtDNN is used during the training loop to provide a differentiable first-order approximation of PQC’s behaviours, local to each mini-batch during the back-propagation The theorem guarantees that replacing the PQC by that MLP changes forward activations by at most ε1 and their Jacobian by at most ε2, which in turn perturbs the gradient flowing into earlier layers by the same order of magnitude This confirms that surrogate-assisted training reduces quantum-specific wall-clock by more than an order of magnitude compared to parameter-shift The hDQNN is evaluated on the Humanoid-v4 benchmark and achieves a final reward score comparable to that of models with widely used SOTA architectures except the recently proposed MEow from Chao et al.

Kai: (2024) The method offers a new route in reinforcement learning and AI research for addressing complicated real-world problems that are always stochastic, highly correlated, and even challenging to simulate classically This work demonstrates the feasibility of surrogate-assisted hybrid quantum training The qtDNN is fitted on the noisy measurements qo at every step, so the gradients used to train the actor are aligned with the behaviour that the policy will experience at deployment The qtDNN's update epoch is split into two stages: Surrogate fit and Network update, where it refits every actor update on the current mini-batch and freezes it only for the remainder of that update epoch The training involves four alternating phases: Exploration, qtDNN update, Critic update (TD learning), and Actor update During the actor update phase, gradients w.r.t.

Paper summary: Kai: To recap, the paper "Training Hybrid Quantum Policies with Classical Surrogates for Actor-Critic Reinforcement Learning" introduces the concept of training hybrid deep quantum neural networks for efficient reinforcement learning. The thesis is that you can use a classical neural surrogate trained on input-output pairs from a replay buffer to approximate the quantum policy locally during backpropagation.

Mira: They argue that this allows batched backpropagation through the classical layers without needing to repeatedly query the actual quantum circuit for gradients, which keeps things compatible with high-throughput GPU pipelines.

Lev: The goal is to provide a mini-batch-local differentiable approximation of the PQC used only in the backward pass, while ensuring that the forward path and inference remain purely quantum. That’s a crucial distinction for hardware deployment.

Kai: They use a specific architecture called hDQNN-TD3, which is based on the TD3 architecture for continuous control reinforcement learning. It matches or exceeds state-of-the-art classical performance on popular benchmarks like Humanoid-v4.

Mira: What they claim is that the qtDNN surrogate provides gradients of sufficient fidelity throughout training, and late stage divergence illustrates how much room there is for the quantum-enhanced actor to exploit once it enters a high performance regime.

Lev: They mention that in the best case, a single well-tuned hDQNN-TD3 seed can match the ceiling performance of the strongest classical baselines while requiring fewer interaction steps to get there. That’s something practical for running on real hardware.

Kai: The qtDNN surrogate is trained on-the-fly from the same minibatch, which accelerates its convergence towards approximating the target quantum layer, similar to what Haarnoja et al. (two thousand eighteen) described in terms of learning on-the-fly <ref:2503.09119#pg1>.

Mira: This allows the qtDNN to learn on-the-fly to emulate the quantum circuit while simultaneously providing the actor with differentiable expectation values, which is key for batched backpropagation through the classical layers without repeated quantum queries for gradients.

Lev: The theorem they provide guarantees that replacing the PQC with that MLP changes forward activations by at most epsilon1 and their Jacobian by at most epsilon2, and this in turn perturbs the gradient flowing into earlier layers by the same order of magnitude. That’s a local gradient-fidelity guarantee.

Conclusion: Kai: So, looking back at "Training Hybrid Quantum Policies with Classical Surrogates for Actor-Critic Reinforcement Learning," we see this method is about making hybrid quantum policy training more practical by using classical surrogates to bypass the need for differentiating the quantum circuit during training.

Mira: It’s about creating a framework where you can get scalable batch gradients while keeping the original quantum layer intact for actual forward interaction and inference.

Lev: The authors demonstrated that this approach, through hDQNN-TD3, can match or exceed state-of-the-art classical performance on benchmarks like Humanoid-v4, which validates the modeling capacity of PQCs in reinforcement learning tasks.

Kai: What this means for us is that we have a new route in reinforcement learning and AI research for tackling complex real world problems that are inherently stochastic and hard to simulate classically.

Mira: It shows the feasibility of surrogate-assisted training as an effective approach to learn hybrid quantum policies, provided you manage the fitting of that classical surrogate correctly, as they show with their gradient error bounds.

Lev: The limitation they point out is that the performance relies on the quality of that local approximation; if it diverges too much during training, you lose fidelity.

Kai: It’s a solid piece of work because it shows how we can use these hybrid quantum models in reinforcement learning tasks that are often challenging to simulate classically.

Anyon Computing Inc. · Airbnb Inc. · Emerging Technologies, Innovation and Ventures, HSBC Holdings Plc., United Kingdom · Quantum Technologies Group, Emerging Technologies, Innovation and Ventures, HSBC Holdings Plc., Singapore

quant-ph

Submitted: 2025-03-12

Updated: 2026-10-19

Code: https://github.com/DLR-RM/rl-baselines3-zoo

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 85/100

The gist: The gist The method introduces qtDNN, a tangential surrogate that locally approximates a PQC with a small differentiable network trained on-the-fly from the same minibatch, which enables scalable

Key concepts

Surrogate-Assisted Actor-Critic: This is a training method that uses a classical neural network (the surrogate) to mimic the behavior of a quantum circuit during the learning process. Instead of trying to calculate gradients through the complex quantum circuit directly, the surrogate provides differentiable approximations for actor updates and critic computations, making backpropagation feasible.
qtDNN: qtDNN is a tangential neural network that acts as a local surrogate for the parametrized quantum circuit. It is trained on-the-fly from the same minibatch used for policy updates. This allows it to provide differentiable gradients to the classical actor while keeping the original quantum layer intact for actual inference.
Hybrid Quantum Policies: These are reinforcement learning policies that combine quantum and classical components. The core idea is that a quantum circuit handles certain parts of the decision-making process, while a classical neural network manages others. This paper focuses on training these combined systems efficiently using surrogate techniques.

Terminology

Quantum Circuits — 2× in this paper · explained in 2 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Actor — 1× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Hilbert Space — 1× in this paper · explained in 2 episodes

Summary

The gist The method introduces qtDNN, a tangential surrogate that locally approximates a PQC with a small differentiable network trained on-the-fly from the same minibatch, which enables scalable batch gradients while keeping the original quantum layer for inference

**The core problem addressed is the prohibitive cost of backpropagating through PQCs to train surrounding classical DNNs This bottleneck makes hybrid training impractical for realistic RL workloads because every gradient step requires a prohibitive number of circuit evaluations and thus erodes the potential quantum advantage Existing gradient estimation techniques make such hybrid training impractical for realistic RL workloads, because every gradient step requires a prohibitive number of circuit evaluations and thus erodes the potential quantum advantage The authors introduce hDQNN-TD3, a hybrid deep quantum neural network for continuous-control reinforcement learning based on the TD3 architecture, which matches or exceeds state-of-the-art classical performance on popular benchmarks This method opens a path toward applying hybrid quantum models to large-scale RL and other gradient-intensive machine learning tasks The goal is to provide a mini-batch-local differentiable approximation of the PQC used only in the backward pass, while the forward path and inference remain quantum This enables scalable mini-batch training by supplying gradients without additional quantum evaluations, compatible with high-throughput GPU pipelines The method offers theoretical support through a local gradient-fidelity guarantee, ensuring that surrogate back-propagation perturbs upstream gradients only within controlled tolerances The empirical effectiveness is demonstrated on the Humanoid-v4 benchmark, outperforming widely adopted classical baselines (Zhao, 2024; 2023a;b) and validating the nontrivial modeling capacity of PQCs compared with fully connected, random, or zero-layer alternatives The hDQNN-TD3 actor matches or exceeds state-of-the-art classical performance on popular benchmarks The qtDNN surrogate provides gradients of sufficient fidelity throughout training, while the late-stage divergence illustrates the head-room the quantum-enhanced actor can exploit once it enters the high-performance regime The results show that in the best case, a single well-tuned hDQNN-TD3 seed can match the ceiling performance of the strongest classical baselines while requiring fewer interaction steps to get there The qtDNN surrogate supplies gradients of sufficient fidelity to stabilize late-stage policy refinement The qtDNN surrogate is trained on-the-fly from the same minibatch, which accelerates qtDNN’s convergence to a moving approximation to its target quantum layer (cf. Haarnoja et al. (2018)) The qtDNN learns on-the-fly to emulate the quantum circuit while simultaneously providing the actor with differentiable expectation values This allows batched back-propagation through the classical layers without repeatedly querying the quantum circuit for gradients The qtDNN is used during the training loop to provide a differentiable first-order approximation of PQC’s behaviours, local to each mini-batch during the back-propagation The theorem guarantees that replacing the PQC by that MLP changes forward activations by at most ε1 and their Jacobian by at most ε2, which in turn perturbs the gradient flowing into earlier layers by the same order of magnitude This confirms that surrogate-assisted training reduces quantum-specific wall-clock by more than an order of magnitude compared to parameter-shift The hDQNN is evaluated on the Humanoid-v4 benchmark and achieves a final reward score comparable to that of models with widely used SOTA architectures except the recently proposed MEow from Chao et al. (2024) The method offers a new route in reinforcement learning and AI research for addressing complicated real-world problems that are always stochastic, highly correlated, and even challenging to simulate classically This work demonstrates the feasibility of surrogate-assisted hybrid quantum training The qtDNN is fitted on the noisy measurements qo at every step, so the gradients used to train the actor are aligned with the behaviour that the policy will experience at deployment The qtDNN's update epoch is split into two stages: Surrogate fit and Network update, where it refits every actor update on the current mini-batch and freezes it only for the remainder of that update epoch The training involves four alternating phases: Exploration, qtDNN update, Critic update (TD learning), and Actor update During the actor update phase, gradients w.r.t. θPre and θPost are computed through the differentiable qtDNN, allowing gradients w.r.

Improvements for AI systems

The hDQNN-TD3 agent can learn complex, high-dimensional, continuous-variable reinforcement learning tasks like Humanoid-v4 successfully with a final reward score comparable to that of models with widely used SOTA architectures except the recently proposed MEow from Chao et al. (2024). This allows for the development of more capable humanoid robots by enabling them to navigate and interact in unpredictable real-world conditions.
The system can achieve sample-efficiency gains, as the hDQNN-TD3 reaches the 3,000-return mark about 800 episodes earlier than its fully connected (FC) alternative, suggesting the quantum layer helps discover high-value policies sooner. This means fewer environmental interactions are required to train a policy that achieves high performance.
The hybrid architecture can be deployed on hardware with noisy quantum operations, as the method demonstrates that hDQNN-TD3 model having PQC bit-flip and phase-flip error rates of 10−3 perform on par and even potentially better than classical SOTA models in terms of rewards they achieved given a similar compute. This opens a path for applying hybrid quantum models to large-scale RL where noisy intermediate-scale quantum (NISQ) devices are the only available hardware.
The system can handle gradient-intensive machine learning tasks efficiently, as the introduction of qtDNN, a tangential surrogate that locally approximates a PQC with a small differentiable network trained on-the-fly from the same minibatch, enables scalable mini-batch training by supplying gradients without additional quantum evaluations. This capability makes hybrid quantum models viable for large-scale RL and other gradient-intensive machine learning tasks.