Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol

arXiv:2610.21741 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol".

Dev: The gist The evaluation protocol for Bench2Drive contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior.

Rosa: First, who's behind it and why it matters.

Paper summary: Dev: So we're wrapping up our look at "Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol." The authors are showing us that how we judge driving models can be flawed if we use the wrong metrics or signal methods #pg1

Rosa: Yeah, it’s about showing that the comfort score you get from Bench2Drive isn't really measuring smooth driving at all because of how they set up the test #pg1

Taro: I mean, they found this thing where standing still gets a high score just because it stays put for a long time in their system #pg6

Dev: Exactly. They showed that stationary time was accounting for ninety three percent of the variance in the traditional score, which is way too much to be true #pg1

Rosa: And they fixed that by using something based on distance instead of just time spent waiting there, cutting that correlation down to zero point zero six #pg8

Taro: So it means you can't tell if a model is comfortable just by looking at the score when they are driving at the same speed, because it's too focused on how fast they go instead of how well they drive #pg1

Dev: That’s what they found. And beyond that, there are these two other problems with how navigation signals get sent to the AI models #pg2

Rosa: Right. First, a default controller can just override the model's own steering and point it toward the route target instead of letting the model do what it wants #pg3

Taro: That means we might miss if a model can actually correct its own drift or react to an obstacle because the system is forcing that angle #pg3

Dev: And second, when they send those navigation signals at evaluation time, that target point might be totally different from what the model actually saw when it was being trained #pg1

Rosa: So this paper suggests that if you want to compare driving models fairly, you gotta check both these things before you even start comparing scores #pg2

Conclusion: Rosa: So we’ve looked at how Bench2Drive tests end-to-end driving models and found that the evaluation setup itself has some serious blind spots, which is what this paper, "Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol," is all about.

Dev: Right. The authors are essentially saying that if you use this specific protocol for judging a model's comfort, the results you get might not actually reflect how well the model drives at all; it’s more about how long it stays still, which is misleading.

Taro: That makes sense because they found that stationary time accounts for ninety-three percent of the variance in their score, which means most of what you see isn't actual driving skill but just sitting there.

Rosa: They fix this by using a distance-based metric instead of looking at time spent, which cuts that dependence down to zero point zero six, showing that comfort trades against how fast the model goes rather than how well it drives.

Dev: And they also pointed out these navigation signal issues where the default controller can override the model's steering and sends a target point at evaluation time that might be totally different from what the AI saw during training.

Taro: So, if you’re building a safety case for an autonomous system, you have to check those two things—the steering overruling and the mismatched target point—before you think the scores are comparable across different models.

Rosa: And that means we need better testing protocols for closed-loop systems in the real world if we want to truly understand how these AI models behave when they're on the road.

Thomas Schwarzfischer, Emad Aghajanzadeh, Moritz Thoma, Nael Fasfous, Manoj Rohit Vemparala, Pierpaolo Mori, Daniel Muller-Gritschneder ®3” and Ulf Schlichtmann2

BMW Group Munich, Germany · Technical University of Munich Munich, Germany · Vienna University of Technology Vienna, Austria

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/motional/nuplan-devkit

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: The gist The evaluation protocol for Bench2Drive contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior.

Key concepts

Comfort Metric Pitfalls: The original Bench2Drive comfort metric had implementation errors and a segmentation scheme that rewarded vehicles staying still. This meant the metric largely measured stationary time instead of smooth driving, leading to misleading comfort scores.
Distance-Based Metric: Proposed as a correction, this new metric partitions the route into equidistant bins along a polyline based on distance traveled rather than time spent. This removes the bias toward standing still and accurately measures comfort based on actual path progression.
Steering Overruling (SO): A pitfall where the default controller replaces a vehicle's steering input with an angle directed toward the route target point. This masks whether a model can independently correct its own drift or react to immediate obstacles.
Target Point Mismatch: The evaluation harness supplies a navigation target point that might differ significantly from what the model experienced during training. This discrepancy introduces out-of-distribution inputs, affecting model performance and comparison.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Steering — 1× in this paper · explained in 2 episodes

Summary

The gist The evaluation protocol for Bench2Drive contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior.

How it works

The paper identifies issues within Bench2Drive’s comfort metric, which is adapted from nuPlan, noting that its implementation errors and segmentation scheme reward standing still rather than smooth driving<ref:2610.21741#pg4>. Specifically, the metric mostly measures how long a vehicle stays stationary, correlating with stationary time at r = 0.97 across eight models<ref:2610.21741#pg4>. The authors propose a corrected, distance-based metric that cuts this correlation to r = 0.06<ref:2610.21741#pg4>. This corrected metric suggests that comfort trades against how fast a model drives rather than how well it drives<ref:2610.21741#pg4>.

Navigation Signals Pitfalls

The paper uncovers two pitfalls in how the harness supplies navigation signals to E2E models<ref:2610.21741#pg4>. First, the default controller can replace a model’s own steering with an angle toward the route target point, a behavior called steering overruling (SO), which masks whether a model can correct its own drift or react to an obstacle<ref:2610.21741#pg5>. Second, the target point supplied at evaluation time can differ substantially from what a model saw in training, pushing it toward out-of-distribution inputs<ref:2610.21741#pg4>. The authors propose fixes for both: a controller that never overwrites the model’s steering and a route planner whose target-point sampling matches the training distribution<ref:2610.21741#pg4>.

Corrected Comfort Metric

The original Bench2Drive comfort metric suffers from several issues, including implementation errors such as sampling-rate mismatch, an under-sized smoothing window, and invalid yaw processing<ref:2610.21741#pg6>. Furthermore, the temporal segmentation rewards standing still by construction because a stationary vehicle satisfies every threshold for free and keeps banking comfortable segments for as long as it stands there<ref:2610.21741#pg6>. To address this, the authors propose a revised metric based on three changes: nuPlan-faithful calculation, smooth-first filtering, and distance-based segmentation<ref:2610.21741#pg6>. The distance-based segmentation partitions each route into N = 100 equidistant bins along the reference polyline, where each bin spans 1% of the route length (∼1 − 2 m)<ref:2610.21741#pg6>. This removes the bias introduced by segments where the vehicle is standing still, and a segment’s importance is tied to distance traveled rather than time spent<ref:2610.21741#pg6>.

Impact of Corrections

The authors demonstrate that fixing the implementation alone does not help because time-based segmentation rewards standing still by construction<ref:2610.21741#pg6>. The corrected metric shows that stationary time explains 93 % of the variance in the traditional score (r = 0.97), while under distance-based segmentation, this dependence is removed, with a correlation of r = 0.06<ref:2610.21741#pg8>. This correction reveals that the strong tradeoff between driving score and comfort is no longer detectable once models are compared at matched driving speed.

Cross-Model Insights

The analysis of the corrected metric shows that the two language-model planners land at opposite extremes of the corrected scale despite sharing that design, pointing to the trajectory decoder rather than the language model as what governs ride quality. The only reinforcement-learned policy in their set rides most smoothly for the speed it drives, without having been trained for comfort at all. The two pitfalls in navigation signal supply affect UniAD and VAD by very different amounts, meaning scores obtained under the default protocol are not comparable across models. Future closed-loop evaluations should therefore check for both before comparing models. The authors conclude that comfort appears to trade against how fast a model drives rather than against how well it drives.

The paper's main contribution is showing that the Bench2Drive protocol itself contains issues that can lead to large misinterpretations of E2E models along two axes: driving comfort and driving behavior. The authors propose a corrected, distance-based metric that removes the bias toward standing still while staying faithful to nuPlan’s thresholds<ref:2610.21741#pg4>. The paper also uncovers two protocol-level pitfalls in how the harness supplies navigation signals, specifically steering overruling and target-point mismatch<ref:2610.21741#pg4>. These findings are crucial because they suggest that comfort trades against how fast a model drives rather than against how well it drives. The paper's main contribution is showing that the Bench2Drive protocol itself contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior. The authors propose a corrected, distance-based metric that removes it while staying faithful to nuPlan’s thresholds<ref:2610.21741#pg4>. The paper also uncovers two pitfalls in how the harness supplies navigation signals, a controller that overwrites the model’s steering and a traindeployment mismatch in the target point, and propose fixes for both<ref:2610.21741#pg4>. These findings are crucial because they suggest that comfort trades against how fast a model drives rather than against how well it drives. The paper's main contribution is showing that the Bench2Drive protocol itself contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior. The authors propose a corrected, distance-based metric that removes it while staying faithful to nuPlan’s thresholds<ref:2610.21741#pg4>. The paper also uncovers two pitfalls in how the harness supplies navigation signals, a controller that overwrites the model’s steering and a traindeployment mismatch in the target point, and propose fixes for both<ref:2610.21741#pg4>. These findings are crucial because they suggest that comfort trades against how fast a model drives rather than against how well it drives. The paper's main contribution is showing that the Bench2Drive protocol itself contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior. The authors propose a corrected, distance-based metric that removes it while staying faithful to nuPlan’s thresholds<ref:2610.21741#pg4>. The paper also uncovers two pitfalls in how the harness supplies navigation signals, a controller that overwrites the model’s steering and a traindeployment mismatch in the target point, and propose fixes for both<ref:2610.21741#pg4>. These findings are crucial because they suggest that comfort trades against how fast a model drives rather than against how well it drives. The paper's main contribution is showing that the Bench2Drive protocol itself contains pitfalls that can misrepresent end-to-end driving models along two axes: driving comfort and driving behavior. The authors propose a corrected, distance-based metric that removes it while staying faithful to nuPlan’s thresholds<ref:2610.21741#pg4>. The paper also uncovers two pitfalls in how the harness supplies navigation signals, a controller that overwrites the model’s steering and a traindeployment mismatch in the target point, and propose fixes for both<ref:2610.21741#pg4>. These findings are crucial because they suggest that comfort trades against how fast a model drives rather than against how well it drives.

Improvements for AI systems

This improvement involves replacing Bench2Drive’s comfort metric with a distance-based metric that cuts that correlation to r = 0.06 and moves VAD and SimLingo sharply down the comfort ranking. This ensures models are scored based on how well they drive per meter driven, rather than how long they stand still, as the original metric was found to be dominated by a bias toward standing still.
The corrected navigation signal supply should involve implementing a controller that never overwrites the model’s steering, which prevents the issue of steering overruling (SO) where the default controller steers toward the route target point instead of respecting the model's own prediction, thereby allowing models to correct its own drift or react to a sudden obstacle.
The evaluation harness must be updated with a route planner that matches the training distribution for navigation targets, and it should use a cursor that only ever moves forward along the route, ensuring the target point progression is continuous instead of jumping, which addresses the target-point mismatch pitfall.