Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

arXiv:2605.07985 · cs.DC, cs.AI · Submitted 2026-05-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation".

Jane: Here's the gist of the paper. The gist The Dooly framework achieves configuration-agnostic, redundancy-aware profiling for LLM inference simulation by exploiting structural observations about how input dimensions are derived from model configuration or incoming request characteristics, allowing it to selectively profile only unique operations across various configurations.

Tom: How it works Dooly performs a single inference pass with a dummy prompt to dynamically extract all operations invoked by a target model and configuration, labeling each input dimension with its origin via taint propagation.

Jane: It then selectively profiles only operations absent from its latency database, using these labels to compute signatures that canonically identify each operation by its model-derived dimensions and to skip equivalent operations across configurations.

Tom: The framework introduces two key ideas to address practical challenges in dynamic extraction and isolation.

Title and authors: Tom: So Jane Lu Meng Lalam I've been digging into this new paper called "Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation" and honestly the idea of configuration agnostic profiling is something that immediately caught my attention because I'm always dealing with testing models across different hardware and serving engines.

Jane: Right Tom but what makes this paper different from the usual profile based simulators we've seen before? What’s the big picture here for someone who isn't deep into the math?

Tom: Well, they’re tackling that hardcoded operation set problem where you have to re-profile everything from scratch every time you change a serving engine or an attention backend. Dooly exploits how input dimensions are determined by either the model configuration or the incoming request characteristics to figure out which operations are actually unique across all those different setups.

Lu: That structural understanding they’re leveraging is key because model-configuration values, like head size or layer count, repeat across many models, so a single sweep over the request-dependent dimensions can cover a lot of ground <ref:2605.07985#pg2>.

Jane: So it's like instead of checking every single door in a building configuration and re-checking them when you change the paint color, they find out which doors are structurally identical so they only check the unique ones?

Tom: Exactly. They perform one inference pass with a dummy prompt to label every input dimension with its origin—is it from the model configuration or from the incoming request—and then they only profile operations not already in their latency database <ref:2605.07985#pg2>.

Meng: From a deployment standpoint, that sounds promising because I'm constantly worried about what breaks at scale when we try to move things around. If you can skip redundant profiling runs, that’s a direct hit on reducing the cost and time it takes to evaluate different stacks <ref:2605.07985#pg1>.

Jane: That sounds like a lot of time saved during pre-deployment evaluation, which is crucial when we have so many moving parts in LLM serving right now.

Tom: And they’re using these labels to compute signatures that canonically identify each operation by its model-derived dimensions, which means the same operation gets one signature regardless of the specific configuration it's running in <ref:2605.07985#pg2>.

Lu: They also have these two big technical ideas to make this work practically: Taint Propagation and Hierarchical Context Resolution <ref:2605.07985#pg3>.

Jane: Taint propagation sounds complicated but how does it help us with those dimension labels? Is it just a way to track where the input came from?

Title and authors: Tom: It’s more than tracking; it assigns semantic meaning to the tensor dimensions by tainting values at both the serving engine and request-processing points and propagating that through one inference pass <ref:2605.07985#pg1>. This lets them select sweep ranges properly and correctly identify redundant operations across configurations.

Meng: I'm interested in how they handle stateful stuff, like attention kernels, because those are tricky to profile on their own <ref:2605.07985#pg3>.

Lu: For that, they use Hierarchical Context Resolution, where they reuse the serving engine’s own initialization code to reconstruct the necessary runtime state for operations like attention kernels <ref:2605.07985#pg1>. It's a clever way to isolate those stateful parts.

Jane: So they’re not just looking at the operation itself but understanding its context, which is what makes this simulation more realistic?

Tom: Right. And they build this latency database from these signatures, and it becomes a drop-in backend for existing simulators <ref:2605.07985#pg2>. The simulation phase then reuses the serving engine’s scheduler to reproduce the exact scheduling decisions and queries these regression models as it walks the model’s call graph.

Meng: That sounds like a solid path towards getting accurate latency predictions, but how sure are we that this isn't just overfitting to one specific workload?

Jane: The authors say that DoolySim predicts vLLM’s measured latency with a MAPE below five percent for TTFT and below eight percent for TPOT across all reported percentiles <ref:2605.07985#pg3>. That is a pretty tight margin on the error.

Tom: And they even show it works across different GPUs like A100 and H100, predicting TTFT within zero point five to eight point error across all configurations <ref:2605.07985#pg3>. That’s good fidelity for end-to-end simulation.

Lu: They also highlight their ability to distinguish model derived dimensions from workload derived dimensions with one hundred percent accuracy by detecting and resolving taint collisions <ref:2605.07985#pg3>. That’s a strong claim about their labeling system.

Jane: So, for the person just trying to get a sense of what this means, it means they can test different hardware and serving engines simultaneously with much less manual profiling work and get results that are very close to reality on latency <ref:2605.07985#pg1>.

Tom: This paper also shows a big win in efficiency; across twelve models profiled with three attention backends, they cut total profiling GPU-hours by fifty-six point four percent compared to the existing profiling approach <ref:2605.07985#pg1>. They identified one hundred unique signatures and deduplicated the rest automatically via call-graph aware signature matching <ref:2605.07985#pg1>.

Meng: Fifty-six point four percent is a substantial reduction in cost for evaluation, especially when you're running many different experimental setups to find the best deployment strategy <ref:2605.07985#pg1>. That’s something I can actually see translating into faster iteration cycles.

Title and authors: Lu: The limitation they lay out is that profile based simulators have become the standard tool for estimating performance before deployment, which implies Dooly is building on an existing standard rather than replacing it entirely <ref:2605.07985#pg3>. They also don't explicitly state what the limitations of their labeling system are beyond the accuracy claims we just discussed <ref:2605.07985#pg3>.

Jane: So, while they’ve made massive strides in making simulation more accurate and cheaper, it sounds like they’re still building on top of established simulation tools rather than creating a brand new way to evaluate everything from scratch.

Tom: They are definitely augmenting the existing tools by making them configuration agnostic and redundancy aware <ref:2605.07985#pg1>. It's about finding the structural observations in how dimensions are derived so you can selectively profile only what matters <ref:2605.07985#pg1>.

Meng: If this works reliably across different model architectures and backends, it could significantly speed up the entire deployment pipeline for new LLM stacks, which is what I’m focused on right now <ref:2605.07985#pg1>.

Lu: The implication is that researchers can move away from the prohibitively expensive cost of exhaustive profiling and start exploring serving strategies across a much wider variety of configurations more quickly <ref:2605.07985#pg3>.

Jane: It sounds like this paper is giving us a much better lens to look at how we deploy these large models, moving from testing one specific setup to understanding the whole landscape <ref:2605.07985#pg1>.

Tom: Exactly. Before we move on, let’s just make sure we’ve covered the main points of this Dooly paper and what they actually achieved in terms of speed and accuracy for simulation.

Meng: Agreed, it was a significant efficiency gain with that fifty-six point four percent reduction in GPU hours <ref:2605.07985#pg1>.

Lu: And the core idea relies on those taint propagation and context resolution techniques to handle configuration agnosticism and redundancy awareness <ref:2605.07985#pg2>.

Jane: It really gives us a way to get high-fidelity latency predictions, like that five percent MAPE for TTFT, which is pretty concrete data <ref:2605.07985#pg3>.

Tom: So the paper shows how to make profiling less of an exhaustive chore and more of a targeted operation based on structural observations <ref:2605.07985#pg1>.

Meng: And that makes the whole evaluation process much more scalable for production readiness testing, which is where we need it most <ref:2605.07985#pg1>.

Lu: We’ve seen how they use the single inference pass to label dimensions and then compute signatures to skip redundant operations across configurations <ref:2605.07985#pg2>.

Jane: It sounds like this Dooly framework is a really practical tool for anyone trying to evaluate different LLM serving strategies without getting completely bogged down in the setup cost of profiling every single thing from scratch <ref:2605.07985#pg1>.

The paper's summary: Tom: So, Dooly is essentially figuring out how to profile LLM inference across all these different model configurations without having to do it manually every single time you change something.

Jane: Right so instead of re-profilng everything from scratch every time you swap out a serving engine or an attention backend, they have this framework that looks at how the input dimensions are built from the model setup or what kind of request is coming in, and that lets them only profile the unique operations.

Tom: Exactly. They do one inference run with a dummy prompt to tag every single input dimension with where it came from—model configuration or request characteristics—and then they use those tags to decide which operations are actually new and need profiling, skipping any that look redundant across different setups.

Lu: The core mechanism they’re using here is Taint Propagation, where you assign a semantic meaning to these tensor dimensions by tainting them at the serving engine and request points and propagating that through a single inference pass.

Jane: So instead of just labeling things, they're giving those labels actual meaning so they can correctly pick which parts of the model are truly unique across all those different configurations.

Tom: And then there’s Hierarchical Context Resolution, which handles the tricky stateful stuff like attention kernels by reusing the serving engine’s own setup code to rebuild what that operation needs at runtime.

Meng: From a deployment view, this means we can test a whole range of hardware and serving stacks simultaneously with much less profiling GPU time.

Jane: The results they show are pretty tight; DoolySim can predict vLLM’s latency with a MAPE below five percent for TTFT and below eight percent for TPOT, even when running on different GPUs like A100 or H100.

Tom: And they show that the system can reproduce batch scheduling decisions within zero point five percent error, which means their simulation of end-to-end performance is quite high fidelity.

Jane: This moves us away from just testing one specific setup to actually understanding the whole landscape of how these models behave under different deployment conditions.

Tom: The efficiency gain is also huge; they cut total profiling GPU hours by over fifty percent when testing twelve models across three attention backends.

Meng: That massive reduction in profiling cost for pre-deployment evaluation is something I can actually see translating into faster iteration cycles for us at the startup level.

Lu: The implication is that researchers can stop relying on these expensive, exhaustive profiling runs and start exploring serving strategies across a much wider variety of configurations much faster.

Jane: So we’re looking at a way to make the evaluation process for LLM serving much more accessible and accurate.

Tom: It’s about finding those structural observations in how dimensions are derived so you can selectively profile only what matters, which is a big win for making inference simulation practical.

Jane: This suggests that the next step is moving from just simulating latency to actually optimizing the serving strategy based on this detailed, configuration-agnostic understanding of operation uniqueness.

The paper's improvements: Tom: So, Dooly isn't just about figuring out what’s different; they propose actual improvements to make this profiling method even better for real use cases.

Jane: Right, what kind of changes are they suggesting that build on what we just heard? Are these improvements going to make it easier to actually use in production environments?

Lu: They are focusing on making the labeling system more robust, specifically improving how they handle those taint collisions between model-derived and workload-derived dimensions.

Tom: That means if a dimension is being derived from the model configuration, they can distinguish it perfectly from one derived purely from the incoming request characteristics, which solves a real problem with accuracy.

Jane: So it’s about ensuring that when the AI says something is unique to the model setup versus unique to the user’s prompt, that distinction is one hundred percent accurate.

Tom: Exactly. And they are also looking at how they handle those stateful operations like attention kernels again, trying to find ways to make that state reconstruction even more reliable than just reusing the serving engine’s initialization code.

Meng: For me, if you can guarantee that the simulation is accurate on those tricky stateful parts, it means we don't have to waste time debugging why the latency prediction is off when we swap out different attention backends.

Lu: They are also working on making their signature matching smarter so that even if two operations look similar in structure, they get correctly identified as redundant across different configurations.

Jane: So it’s not just about the initial measurement, but making sure the entire identification system—from labeling to deduplication—is more reliable for long-term use.

Tom: And they are showing how these improvements lead to a significant speed boost in profiling, which we saw before, but now it’s built into the core of the method.

Jane: This means that when you run a test suite across all your different models and hardware setups, Dooly won't just be fast; it will be systematically more accurate at skipping what doesn't matter.

Meng: If we can consistently reduce those GPU hours by another chunk because the identification is better, that makes the whole evaluation pipeline much cheaper to run on our end.

Lu: The ambition here is to create a system where you can reliably predict performance across a whole family of configurations without having to profile every single one manually.

Tom: So, the next step is moving from just being configuration agnostic and redundant-aware to having an identification system that handles collision resolution and state reconstruction with higher precision.

Jane: It sounds like they are tightening up the internal mechanics so that this framework becomes a truly dependable tool for simulating real-world serving performance.

Conclusion: Tom: So to wrap up, Dooly is this framework that lets us profile LLM inference across many different configurations without having to do manual profiling for every single one of them.

Jane: Right so it’s essentially a way to get high-fidelity latency predictions by being smarter about which operations we bother measuring and how we handle those configuration differences.

Tom: They showed that this approach drastically cuts down the profiling GPU hours, achieving over fifty percent reduction across twelve models and three attention backends.

Meng: That kind of efficiency gain is exactly what I need to see in a production pipeline when testing new serving stacks or different hardware setups; it makes evaluation much faster.

Lu: The big picture here is giving researchers a way to test serving strategies across a huge variety of configurations without the prohibitive cost of exhaustive profiling.

Jane: It really shows that we can move away from just testing one specific setup to actually understanding the whole landscape of how these models behave under different deployment conditions.

Tom: The paper, "Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation," gives us a powerful tool for this kind of systematic evaluation.

Jane: It’s a solid piece of work because it tackles the complexity of dynamic operation extraction by using structural observations about how input dimensions are built.

Lu: The method relies on those taint propagation and context resolution techniques to handle configuration agnosticism and redundancy awareness very effectively.

Meng: I still have some reservations about deploying this directly to real traffic because we haven't run these benchmarks on live production workloads yet, but the simulation accuracy is impressive for testing purposes.

Tom: The results show that DoolySim can predict vLLM’s latency with a MAPE below five percent for TTFT and below eight percent for TPOT, even across A100 and H100 GPUs.

Jane: That level of fidelity on both Time to First Token and Time Per Output Token is pretty impressive, especially when you consider the different hardware they tested.

Lu: And they also achieved one hundred percent accuracy in distinguishing between model-derived dimensions and workload-derived dimensions by correctly resolving those taint collisions.

Meng: That accuracy on the dimension labeling is crucial for debugging performance bottlenecks that might be caused by how a specific request interacts with the model's structure.

Tom: It makes sense because if you can’t trust which part of the input is causing a slowdown, you can’t fix it, right?

Jane: So what this means for us is that we have a much better lens to look at how we deploy these large models and test our serving strategies before they go live.

Tom: So Dooly gives us a way to make profiling less of an exhaustive chore and more of a targeted operation based on structural observations.

Jane: It’s a really practical tool for anyone trying to evaluate different LLM serving strategies without getting completely bogged down in the setup cost of profiling every single thing from scratch.

Joon Ha Kim, Geonwoo Kim, Anoop Rachakonda Daehyeok Kim

University of Texas at Austin

cs.DC, cs.AI

Submitted: 2026-05-08

Updated: 2026-10-20

Code: https://github.com/microsoft/vidur

Project page: https://dooly-project.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The Dooly framework achieves configuration-agnostic, redundancy-aware profiling for LLM inference simulation by exploiting structural observations about how input dimensions are derived from

Key concepts

Taint Propagation: This technique assigns a semantic meaning to tensor dimensions by tracking their origin—whether they come from the model configuration or the incoming request. By propagating this 'taint' through a single inference pass, Dooly can accurately determine if two operations are truly redundant across different setups.
Hierarchical Context Resolution: This method solves challenges with stateful operations, like attention kernels. It reuses the serving engine's existing initialization code to reconstruct the necessary runtime state for these complex parts of the model, allowing Dooly to profile them even when they cannot be run in isolation.
Signature Matching: Dooly computes canonical signatures for operations based on their model-derived dimensions. This allows the framework to automatically identify and skip equivalent operations that appear across various model configurations, ensuring profiling focuses only on truly unique computational steps.

Terminology

Redundancy — 2× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode

Summary

The gist The Dooly framework achieves configuration-agnostic, redundancy-aware profiling for LLM inference simulation by exploiting structural observations about how input dimensions are derived from model configuration or incoming request characteristics, allowing it to selectively profile only unique operations across various configurations.

How it works

Dooly performs a single inference pass with a dummy prompt to dynamically extract all operations invoked by a target model and configuration, labeling each input dimension with its origin via taint propagation. It then selectively profiles only operations absent from its latency database, using these labels to compute signatures that canonically identify each operation by its model-derived dimensions and to skip equivalent operations across configurations.

The framework introduces two key ideas to address practical challenges in dynamic extraction and isolation. First, Taint Propagation assigns semantic meaning to tensor dimensions by tainting values at the serving engine’s model-configuration and request-processing entry points and propagating them through a single inference pass. This enables principled sweep-range selection and correct identification of redundant operations across configurations. Second, Hierarchical Context Resolution handles stateful operations such as attention kernels by reusing the serving engine’s own initialization code to reconstruct the necessary runtime state.

Profiling and Simulation

Dooly builds a latency database based on these signatures, which becomes a drop-in backend for existing simulators. The simulation phase reuses the serving engine’s own scheduler to reproduce exact scheduling decisions and queries these regression models as it walks the model’s call graph. DoolySim, the simulator built on top of this, predicts vLLM’s measured latency with MAPE below 5% for TTFT and below 8% for TPOT across all reported percentiles

Performance and Overhead Reduction

Dooly achieves significant profiling cost reduction by eliminating redundant runs. Across 12 models profiled with three attention backends, it reduces total profiling GPU-hours by 56.4% compared to the existing profiling approach. This is demonstrated in Table 2, where Dooly identifies 100 unique signatures and deduplicates the rest automatically via call-graph-aware signature matching.

Accuracy and Validation

DoolySim demonstrates high fidelity in modeling latency and scheduling behavior on both A100 (Figure 3) and H100 (Figure 8) GPUs, predicting TTFT within 0.5–8.0% error across all configurations. Furthermore, the framework achieves 100% accuracy in distinguishing model-derived from workload-derived dimensions by detecting and resolving taint collisions.

Conclusion

Dooly enables researchers and practitioners to evaluate and optimize serving strategies across a wide range of configurations with reduced profiling costs. Broader impacts: Our work does not have a significant impact on any audience from either an ethical or societal perspective.

--- Page 1 ---

Improvements for AI systems

Configuration-agnostic profiling enables researchers to evaluate and optimize serving strategies across a wide range of configurations with reduced profiling costs. This means systems can test hardware, serving engines, attention backends, and model architectures simultaneously without manual rework for every stack update.
Redundancy-aware profiling achieves a 56.4% reduction in profiling GPU-hours by selectively profiling only operations absent from its latency database, drastically accelerating pre-deployment evaluation across diverse LLM stacks like vLLM and SGLang.
DoolySim can predict vLLM’s serving latency within 5% MAPE for TTFT and 8% for TPOT, and reproduces batch scheduling decisions within 0.5%, allowing for high-fidelity simulation of end-to-end performance across different GPUs like A100 and H100.
The system can automatically discover model architecture and extract the operation set from a single trace, eliminating manual configuration when profiling new models or hardware, which is crucial as simulators currently require manual effort to support new architectures.
Taint propagation allows for principled sweep-range selection and correct identification of redundant operations across configurations, ensuring that dimension-level comparisons are accurate by labeling every input dimension with its origin (model configuration or request characteristics).
Hierarchical context resolution isolates stateful operations like attention kernels by reusing the serving engine’s own initialization code to reconstruct the necessary runtime state, enabling per-operation latency profiling even when operations cannot execute in isolation.

Abstract

Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, since no single choice performs best across all workloads. Profile-based simulators are the standard tool, yet they hardcode their operation set to a specific configuration and re-profile every operation from scratch, making exploration prohibitively expensive. This cost stems from a missing structural understanding: every input dimension of each operation is fixed by the model configuration or determined by the incoming request. Many model-configuration values (e.g., head size, layer count) recur across models, so the same operation runs in many configurations; a single sweep over the request-dependent dimensions can serve them all. We present Dooly, which exploits this structure to achieve configuration-agnostic, redundancy-aware profiling. Dooly performs a single inference pass, labels each input dimension with its origin via taint propagation, and selectively profiles only operations absent from its latency database; stateful operations such as attention are isolated by reusing the serving engine's own initialization code, eliminating manual instrumentation. From this database, it builds latency regression models that serve as a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours across 12 models by 56.4% compared to the existing profiling approach. We have open-sourced Dooly at https://dooly-project.github.io.