Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control

arXiv:2508.20784 · cs.AI · Submitted 2025-08-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control".

Jane: Here's the gist of the paper. The gist Single-agent robust deep reinforcement learning for bus fleet control proposes a framework to mitigate bus bunching in realistic, bidirectional timetabled transit systems by transforming the multi-agent problem into a single-agent formulation using categorical identifiers and a structured reward function.

Tom: Problem Context Bus bunching remains a pervasive challenge in contemporary urban public transportation systems, occurring when two or more buses operating on the same route cluster together, thereby compromising service reliability and prolonging passenger waiting times <ref:2508.20784#pg2>.

Jane: This phenomenon is driven by the dynamic interplay between traffic conditions and passenger demand <ref:2508.20784#pg4>.

Tom: In realistic, bidirectional, and timetable-governed networks, multi-agent reinforcement learning (MARL) struggles due to severe data imbalance across agents and structural differences like finite trip structures <ref:2508.20784#pg7>.

Paper summary: Tom: We’re talking about the paper "Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control" by Yifan Zhang Liang Zheng and collaborators here. It tackles bus bunching in urban transit systems that have real-world complexities like different routes and fluctuating demand.

Jane: That sounds really specific. So, what is the main idea behind this research? What problem are they trying to solve with this single agent reinforcement learning framework?

Tom: The paper proposes a robust single-agent reinforcement learning framework for bus holding control in a bidirectional timetabled bus line <ref:2508.20784#pg1>. It’s explicitly designed to avoid the data imbalance and convergence issues that often plague multi-agent reinforcement learning, especially in realistic simulations <ref:2508.20784#pg1>.

Lu: What they do is transform the inherently multi-agent problem into a single-agent one by augmenting the state space with categorical identifiers like vehicle ID, station ID, and time period alongside numerical features such as headway, occupancy, and segment velocity <ref:2508.20784#pg1>. That’s how they capture those inter-agent dependencies without needing a massive multi-agent setup.

Meng: I'm interested in the practical side here. If you have to encode vehicle IDs and station IDs into every single state, how much computational overhead is that? What breaks at scale when you try to deploy that kind of representation?

Tom: They are using a bidirectional timetabled network with dynamic passenger demand in their construction <ref:2508.20784#pg1>. The key innovation is this high-dimensional encoding, which lets the single agent policy capture those dependencies, analogous to projecting non-separable inputs into a higher-dimensional space <ref:2508.20784#pg1>.

Jane: So, they’re trying to make one agent smart enough to handle the coordination that usually requires many separate agents working together. But how do they actually train this single policy? What kind of learning algorithm are they using?

Tom: They employ an event-driven variant of the soft actor-critic, SAC algorithm <ref:2508.20784#pg3>. SAC optimizes an objective function that balances the reward and entropy to encourage both exploration and robustness <ref:2508.20784#pg3>.

Lu: And they’ve also designed a structured reward function that is really interesting. Instead of just giving huge penalties for deviations in headway, they use something called a ridge-shaped reward <ref:2508.20784#pg3>. This balances uniform headways with schedule adherence simultaneously <ref:2508.20784#pg3>.

Meng: A ridge-shaped reward sounds like a good way to incentivize both goals at once. But what about the comparison against other methods? Are they showing that this single agent approach is actually better than the multi-agent baselines?

Tom: Yes, they show that Embedding-SAC achieves more stable and superior performance compared to benchmarks like MADDPG <ref:2508.20784#pg1>. They report a control policy that eliminates all bunching events across the entire simulation period <ref:2508.20784#pg1>.

Jane: So, for someone just driving or walking, what does that mean in plain terms? It means the system manages those bus delays much better than systems using traditional multi-agent methods under noisy conditions.

Lu: It validates that these rich categorical representations are essential for capturing the temporal dynamics of bus operations in these heterogeneous environments <ref:2508.20784#pg1>. That structure allows it to generalize across different routes without needing a totally new model for every single bus <ref:2508.20784#pg1>.

Meng: I’m still worried about the deployment costs and the stability when you move from simulation to actual live traffic data. If we feed it real-world noise, how much does that stability drop off?

Tom: The paper shows SAC’s consistent superiority across all noise levels <ref:2508.20784#pg1>. It outperforms both TD3 and DDPG by a substantial margin under all testing conditions <ref:2508.20784#pg1>.

Jane: So, to put it simply, this work suggests that we can move away from complex multi-agent coordination models and toward simpler, more robust single-agent AI when the state is properly structured <ref:2508.20784#pg1>.

Lu: It confirms that those categorical identifiers are vital for handling the temporal dynamics of bus operations in environments where things aren't perfectly uniform <ref:2508.20784#pg1>.

Meng: It’s a solid result on their simulation, but I still think we need to see how stable this runs when you feed it actual live traffic data instead of a perfect simulator <ref:2508.20784#pg3>.

Tom: Exactly, that’s what we gotta look at next. So, the question is how does this structure hold up when the reality gets messy and unpredictable <ref:2508.20784#pg3>.

Conclusion: Tom: So we’re wrapping up on "Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control." This paper tackles bus bunching in real-world transit without needing a massive multi-agent setup.

Jane: I mean, the title is pretty straightforward, right? Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control. So it’s taking a problem that usually needs lots of different agents and trying to solve it with just one smart agent.

Lu: Yeah, the authors are working on this by taking a multi-agent problem and reformulating it into a single agent one by adding specific identifiers to the state space. That's where they do the heavy lifting with that categorical structuring.

Meng: I’m wondering about the real-world deployment here. If you have to encode vehicle IDs and station IDs into every single state, how much computational overhead is that? What breaks at scale?

Tom: Well, the paper points out it uses a soft actor-critic algorithm, SAC. And they claim this version achieves better stability and performance than the MADDPG baselines we've seen in simulations <ref:2508.20784#pg1>.

Jane: So what does that mean for the person just listening? It means that instead of having a bunch of separate cars trying to coordinate, you have one central brain managing the whole fleet.

Lu: Exactly. The authors show that this single agent can capture those inter-agent dependencies through this higher-dimensional encoding, which lets it generalize across different routes without needing a totally new model for every single bus <ref:2508.20784#pg1>.

Meng: It’s interesting that they emphasize the reward function design too. They aren't just punishing schedule misses with huge penalties; they use something called a ridge-shaped reward to balance uniform headways and sticking to the timetable at the same time <ref:2508.20784#pg1>.

Tom: And when we look at the results, Embedding-SAC eliminates all bunching events in their simulation across all noise levels. That’s a pretty clean result if you ask me <ref:2508.20784#pg1>.

Jane: So, to put it simply, this work suggests that we can move away from complex multi-agent coordination models and toward simpler, more robust single-agent AI when the state is properly structured.

Lu: It validates that rich categorical representations are essential for capturing the temporal dynamics of bus operations in these heterogeneous environments <ref:2508.20784#pg1>.

Meng: It’s a solid result on their simulation, but I still think we need to see how stable this runs when you feed it actual live traffic data instead of a perfect simulator.

Tom: Exactly, that’s what we gotta look at next. So the question is how does this structure hold up when the reality gets messy and unpredictable?

Yifan Zhang, Liang Zheng

Central South University

cs.AI

Submitted: 2025-08-28

Updated: 2026-10-17

DOI: 10.1093/tse/tdag005

Code: https://github.com/erzhu419/Categorical-Feature-sac-in-bus-simulation

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist Single-agent robust deep reinforcement learning for bus fleet control proposes a framework to mitigate bus bunching in realistic, bidirectional timetabled transit systems by transforming the

Key concepts

Bus Bunching
This occurs when multiple buses on the same route cluster together, causing delays and long waits for passengers. It is caused by unpredictable traffic and fluctuating passenger demand in urban transit networks.
State Space Augmentation
The paper enhances the input state by adding categorical identifiers like vehicle ID and station ID alongside numerical data like speed. This allows a single agent to understand the specific context of each bus and location, making the problem solvable even with complex, real-world routes.
Ridge-Shaped Reward Function
The reward system is designed not just to penalize delays but to balance two goals: keeping headways uniform and sticking closely to the schedule. This 'ridge' shape ensures the learned policy prioritizes both smooth service intervals and timely departures.
Soft Actor-Critic (SAC)
SAC is a specific deep learning algorithm used for reinforcement learning that optimizes a trade-off between maximizing rewards and maintaining high exploration (entropy). It is used here because it provides stable performance, helping the agent learn robust policies even when facing unpredictable traffic conditions.

Terminology

Summary

The gist Single-agent robust deep reinforcement learning for bus fleet control proposes a framework to mitigate bus bunching in realistic, bidirectional timetabled transit systems by transforming the multi-agent problem into a single-agent formulation using categorical identifiers and a structured reward function.

Problem Context

Bus bunching remains a pervasive challenge in contemporary urban public transportation systems, occurring when two or more buses operating on the same route cluster together, thereby compromising service reliability and prolonging passenger waiting times <ref:2508.20784#pg2>. This phenomenon is driven by the dynamic interplay between traffic conditions and passenger demand <ref:2508.20784#pg4>. In realistic, bidirectional, and timetable-governed networks, multi-agent reinforcement learning (MARL) struggles due to severe data imbalance across agents and structural differences like finite trip structures <ref:2508.20784#pg7>.

Proposed Framework

The study proposes a robust single-agent deep reinforcement learning (RL) framework explicitly designed to circumvent the data imbalance and convergence issues associated with MARL <ref:2508.20784#pg2>. The key contribution involves transforming the inherently multi-agent problem into a single-agent formulation by explicitly encoding categorical identifiers—such as vehicle, station, and trip IDs alongside traditional numerical features—together as the state representation <ref:2508.20784#pg2>. This feature space augmentation enables the single-agent to operate effectively in a higher-dimensional space, analogous to projecting linearly inseparable inputs into a higher-dimensional space to achieve separability <ref:2508.20784#pg2>.

State Representation and Action Space

The state representation s includes both categorical and numerical features, where the categorical component consists of [bus id, station id, time period, direction] encoded through learnable embedding layers <ref:2508.20784#pg10>. These are concatenated with numerical features such as [forward headway, backward headway, current segment speed] to form the final input vector s <ref:2508.20784#pg10>. The action definition corresponds to holding time at the current station, defined as a scalar value sampled from a bounded continuous space [0, Hmax], which is chosen over speed control due to safety and operational constraints.

Reward Function Design

The reward function is designed to promote both symmetry in headways and schedule adherence. It consists of three components: Schedule alignment, which is a soft penalty proportional to the deviation from the nominal target headway τ, Headway symmetry, a similarity bonus inversely proportional to the absolute difference between hf and hb, and Robust penalization. This design results in a ridge-shaped reward function that incentivizes alignment with both uniform headways and scheduled departure intervals.

Algorithm and Robustness

The proposed algorithm utilizes an event-driven variant of the soft actor-critic (SAC) algorithm. SAC optimizes the objective J(π) = Σ E(st,at)∼ρπ [r(st, at) + αH(π(·st))] (10), which balances reward and entropy to encourage exploration and robustness. The use of categorical embeddings allows the single-agent framework to generalize across heterogeneous spatio-temporal contexts without requiring separate parameterizations for each vehicle instance. Extensive experiments confirm that SAC with full categorical embeddings achieves the best performance in every testing variance setting, demonstrating its superior robustness under stochastic conditions.

Experimental Results

Empirical results demonstrate that Embedding-SAC achieves better performance in terms of cumulative mean reward value, asymptotic stability, and variance of cumulative reward value compared to both parameter-sharing and non-sharing variants of MADDPG. Specifically, the control policy eliminates all bunching events across the entire simulation period, as confirmed by trajectory-level visualizations and spatio-temporal heatmaps. SAC’s consistent superiority across all noise levels is evident as it outperforms both TD3 and DDPG by a substantial margin under all testing conditions. Furthermore, the full embedding architecture achieves the best reward in every σtest setting, confirming that rich categorical representations are essential for capturing the heterogeneity and temporal dynamics of bus operations.

Conclusion

The paper concludes that Embedding-SAC achieves better performance in terms of cumulative mean reward value, asymptotic stability, and variance of cumulative reward value compared to both parameter-sharing and non-sharing variants of MADDPG. This validates that a wellstructured single-agent policy, when leveraging appropriate embedded features, can effectively optimize holding decisions under uncertainty >. The work demonstrates that SAC with rich categorical embeddings uniquely combines high performance, robustness to variance shifts, and stable cross-domain generalization properties critical for real-world transit systems where demand and travel times vary unpredictably.

--- Page 1 ---

The gist Single-agent robust deep reinforcement learning for bus fleet control proposes a framework to mitigate bus bunching in realistic, bidirectional timetabled transit systems by transforming the multi-agent problem into a single-agent formulation using categorical identifiers and a structured reward function.

Algorithm and Robustness

The proposed algorithm utilizes an event-driven variant of the soft actor-critic (SAC) algorithm. SAC optimizes the objective J(π) = Σ E(st,at)∼ρπ [r(st, at) + αH(π(·st))] (10), which balances reward and entropy to encourage exploration and robustness. The use of categorical embeddings allows the single-agent framework to generalize across heterogeneous spatio-temporal contexts without requiring separate parameterizations for each vehicle instance.

Experimental Results

Empirical results demonstrate that Embedding-SAC achieves better performance in terms of cumulative mean reward value, asymptotic stability, and variance of cumulative reward value compared to both parameter-sharing and non-sharing variants of MADDPG. Specifically, the control policy eliminates all bunching events across the entire simulation period, as confirmed by trajectory-level visualizations and spatio-temporal heatmaps. SAC’s consistent superiority across all noise levels is evident as it outperforms both TD3 and DDPG by a substantial margin under all testing conditions.

Improvements for AI systems

  1. Integration of categorical identifiers into state representation allows a single-agent policy to generalize across heterogeneous agents by mapping discrete identifiers like bus ID, station ID, and trip ID into dense latent representations. This enables the agent to generalize across heterogeneous buses and temporal contexts without needing separate parameterizations for each vehicle instance.

  2. Implementation of a ridge-shaped reward function incentivizes both uniform headways and scheduled departure intervals simultaneously. This design ensures the agent is incentivized to align with both uniform headways and scheduled departure intervals, leading to more stable control by promoting platoon balance along the line hf = hb = τ.

  3. Utilizing a soft actor-critic framework with an entropy regularization term enables the policy to remain stochastic during training, which enhances exploration and naturally promotes more robust behavior in environments characterized by noise or partial observability. This mechanism helps SAC tolerate dynamics variability and ensures the learned policy does not collapse to greedy behavior too early.

  4. The event-driven RL framework, where decisions are triggered upon completion of passenger activities at a stop, ensures that the reward reflects the realized headway regularity by only committing transitions to the replay buffer when a vehicle reaches its subsequent stop. This prevents temporal leakage and provides stable training in partially observable and event-driven environments.

  5. The state representation concatenates categorical embeddings with numerical features like forward headway, backward headway, and current segment speed to create a hybrid structure that allows the policy network to incorporate heterogeneous information sources. This explicitly addresses data imbalance by allowing the model to learn from experiences of buses operating during different temporal contexts.

Abstract

Bus bunching remains a challenge for urban transit due to stochastic traffic and passenger demand. Traditional solutions rely on multi-agent reinforcement learning (MARL) in loop-line settings, which overlook realistic operations characterized by heterogeneous routes, timetables, fluctuating demand, and varying fleet sizes. We propose a novel single-agent reinforcement learning (RL) framework for bus holding control that avoids the data imbalance and convergence issues of MARL under near-realistic simulation. A bidirectional timetabled network with dynamic passenger demand is constructed. The key innovation is reformulating the multi-agent problem into a single-agent one by augmenting the state space with categorical identifiers (vehicle ID, station ID, time period) in addition to numerical features (headway, occupancy, velocity). This high-dimensional encoding enables single-agent policies to capture inter-agent dependencies, analogous to projecting non-separable inputs into a higher-dimensional space. We further design a structured reward function aligned with operational goals: instead of exponential penalties on headway deviations, a ridge-shaped reward balances uniform headways and schedule adherence. Experiments show that our modified soft actor-critic (SAC) achieves more stable and superior performance than benchmarks, including MADDPG (e.g., -430k vs. -530k under stochastic conditions). These results demonstrate that single-agent deep RL, when enhanced with categorical structuring and schedule-aware rewards, can effectively manage bus holding in non-loop, real-world contexts. This paradigm offers a robust, scalable alternative to MARL frameworks, particularly where agent-specific experiences are imbalanced.

Sources

Related papers