TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces

arXiv:2605.29941 · cs.NI, cs.LG · Submitted 2026-05-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces".

Jane: The gist The core problem is the decode interface, which mixes behavioral choices with protocol consequences,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper now, "TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces," and the big idea here is solving a real problem in creating high-fidelity network data.

Jane: Exactly. The core issue they point out is that when you use existing tools to synthesize these packet captures, you end up with something where the learned behavior and the strict rules of the network protocol get mixed up.

Lu: It seems they see this as a bottleneck at the decode interface, where a model tries to learn what's happening while also having to follow deterministic protocol consequences.

Meng: So instead of just learning raw packet fields, they propose something that separates those learned behavioral choices from the actual protocol rules that determine how packets should look.

Tom: Right, and their proposed solution is this state-aware neural codec called TRACECODEC which takes a packet and turns it into a timed packet action with explicit flow slots and transport cues.

Jane: That action record has all the behavioral choices—like what the endpoint decides to do—and that's mapped into a continuous latent vector.

Lu: Then, this learned component decodes that vector back into those packet action fields, which are then fed to a deterministic compiler for final realization.

Meng: The key here is that this compiler handles all the tricky stuff like assigning endpoint templates and tracking TCP state evolution deterministically.

Tom: It means the system isn't just guessing the final packet; it’s building it from decoded actions under explicit flow state, which really locks down the legality of what comes out.

Jane: The paper shows that this approach reproduces packet count, inter-arrival time, and flow count within one percent of real traces on something called CICIDS2017 Monday.

Lu: They also highlight that their method preserves TCP state transitions and multi-flow interleaving in a way that raw field decoders fragment into different pieces.

Meng: But they found some trade-offs too, which is important for practical use. They noted that while the reconstruction error is very low on count and protocol composition, there's still session-duration distortion around nine point two three percent.

Tom: That distortion comes from the fact that even with this state-aware compiler, the state transitions are a consequence of the decoded actions under explicit transport state, not just a set of independently decoded raw fields after the fact.

Jane: So what they're suggesting is that we should be building traffic generators around this explicit state-aware decoding rather than relying on some kind of heuristic packet repair afterward.

Lu: Their latent interface is designed specifically to be a usable surface for imperfect downstream generators, which means other models can operate on these packet-action latents instead of raw header fields.

Meng: From an engineering standpoint, the privacy aspect is also something they addressed by suppressing direct identifiers like raw IP addresses and application payload bytes.

Title and authors: Tom: They do this by re-instantiating synthetic endpoints based on flow-slot identity, which keeps the necessary packet and transport-level semantics while reducing identifying information.

Jane: The paper mentions that their full model reconstructs packet actions at a ninety-nine point seven three percent accuracy and keeps decoded count, inter-arrival time, and flow count errors below one percent.

Lu: They also tested how removing fine timing affects things; they found that dropping it causes the inter-arrival time distortion to jump from zero point five nine percent up to thirteen point three seven percent.

Meng: That's a big warning for anyone trying to use this for precise simulations, showing that packet timing has a real impact on fidelity.

Tom: It really suggests that if you want high-fidelity traffic generation, you need to be very explicit about the state information being passed through the decoding process.

Jane: The overall implication is a practical path toward higher-fidelity traffic generators built around explicit state-aware decoding instead of just patching up raw data after it's been generated.

Lu: It moves us away from that brittle failure mode where things depend on post-hoc heuristic repair, which was the fundamental problem they set out to solve with this TraceCodec.

Meng: This approach seems much more robust for generating traffic that actually respects protocol constraints from the start, which is what security analysis needs.

Tom: So to wrap up this discussion on "TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces," they've shown how to separate learned behavior from deterministic consequences using a state-aware compiler.

Jane: It’s about taking the packet and turning it into a timed packet action, decoding that into a latent space, and then having a compiler realize those actions as state-consistent packets.

Lu: This provides a clean contract for downstream sequence models to use, exposing what they call a generator-facing sequence space.

Meng: The results show very low error rates on count and protocol composition when compared to raw field baselines, but they did flag the issue with session duration distortion being around ninety two point three percent.

Tom: So while it's a big step in making traffic generation more reliable by focusing on state, you still have to be careful about those timing details if you want perfect accuracy.

Jane: It’s a solid piece of research because it gives us a clear direction for how to structure future tools that need high-fidelity network data for things like security testing.

Lu: I think the way they frame the problem—identifying that decode interface as the fundamental bottleneck—really sets up this whole architecture.

Meng: For practical application, it means we don't just have to fix errors later; we have to build the decoding structure in a way that respects how protocols actually work.

Tom: So if you're building something to test network systems, thinking about explicit state management during generation sounds like the right direction for high-quality results.

Jane: That’s the gist of it—using TRACECODEC to move beyond just statistical flow summaries and toward actual, state-consistent packet traces.

The paper's summary: Tom: So, we're looking at the summary of TRACECODEC now, and what it really boils down to is this separation between how a computer *thinks* about a packet and how that packet actually *looks*.

Jane: Exactly. They're saying that instead of having one messy interface where you put all your learned stuff and all your protocol rules together, they split it up.

Lu: Think of it like this: first, you take a packet and turn it into a structured packet action. This action has the timing, the flow context—all the behavioral choices.

Meng: And then, they use a learned component to map that entire thing into a continuous latent vector. That's the learned interface we're talking about for sequence models.

Tom: Right, and then, this deterministic compiler takes that latent code and lowers it back down into concrete packets. But here’s the important part: the compiler owns all the hard stuff like flow slot assignment and TCP state evolution.

Jane: So it means the final packet isn't just some guess; it has to follow these explicit transport rules dictated by that compiler, not just whatever random bits came out of the decoder.

Lu: The key finding they show is that this structured path—from action to latent code to state-consistent packet—is almost a perfect symbolic match. It’s almost lossless in terms of count and protocol composition on things like the CICIDS2017 Monday dataset.

Meng: That’s impressive because usually, when you just decode raw fields, you get flow counts and TCP states that are completely scrambled after the fact. This system keeps them together throughout the process.

Tom: But there's a catch they mention too; while it’s great for structure, if you remove those fine timing details from the model, your inter-arrival time distortion jumps pretty high—from zero point five nine percent up to thirteen point three seven percent.

Jane: So what this means for us listening right now is that if you're building a traffic generator, you have to be very deliberate about passing on those timing signals.

Lu: And they also addressed privacy by intentionally dropping raw IP addresses and application payload bytes in the learned representation, replacing them with flow-slot identities instead.

Meng: That’s smart because it lets us analyze the behavior without exposing identifying information directly in the data we generate.

Tom: This whole setup suggests a new direction for building traffic generators—focusing on this state-aware decoding rather than trying to fix errors after they happen.

Jane: It really moves us toward generating traffic that actually respects protocol constraints from the start, which is crucial for testing security systems.

Lu: So, the next thing we should look at is how these latent representations are actually being used by downstream sequence models when they try to match real traces.

The paper's improvements: Tom: So, we're looking at how TRACECODEC actually improves things compared to older methods, and what that means for building these traffic simulators.

Jane: Basically, they show that by using this state-aware compiler to finalize the packet realization, you get a much cleaner contract for the downstream sequence models to work with.

Lu: The biggest improvement is moving away from post-hoc heuristic repair. Instead of patching up messy raw fields after generation, you have a deterministic process that owns endpoint assignment and TCP state directly.

Meng: That means we can trust the output more because the state transitions, like SYN to SYN-ACK, are preserved exactly as they should be in a real network flow.

Tom: Right, and they’re showing this latent interface is usable even for imperfect downstream generators. It lets those models operate on these packet-action latents instead of just raw header fields.

Jane: That makes the whole process more robust because the model doesn't have to perfectly decode every single bit of a raw packet; it just needs to match that structured latent space.

Lu: They also found that this approach helps in privacy by suppressing stuff like raw IP addresses and port numbers, replacing them with synthetic endpoint assignments based on flow slot identity.

Meng: It’s a practical win because we get the necessary traffic semantics without exposing sensitive identifying data in the generated traces.

Tom: And they did show their performance metrics on CICIDS2017 Monday, where they matched packet counts and protocol compositions to within zero point zero three percent of the real data. That’s a pretty tight margin for this kind of simulation work.

Jane: That number is strong because it shows that even with this complex system, you still get very high fidelity on the basic structures we care about most, like flow counts and protocol mixes.

Lu: But they did flag one limitation: removing the fine timing information from the model really hurts accuracy in inter-arrival time, pushing that distortion way up to thirteen point three seven percent.

Meng: So if you need perfect timing for a simulation, you can't just throw away those small timing signals; they’re actually important for fidelity.

Tom: It also showed that even if you remove TCP state cues entirely, the action accuracy stays pretty high at ninety-nine point nine five percent, but the protocol event distance gets much worse.

Jane: So what this changes for us is a clear path: focus on building traffic generators around explicit state-aware decoding instead of just hoping a post-processing repair will fix the errors later.

Lu: The next step they’re hinting at is training conditional flow-matching backbones using those latent prefixes and suffixes to test whether models can actually produce the right sequence of actions.

Conclusion: Tom: So we’re wrapping up on "TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces," and the main takeaway is that they’ve managed to turn learned behavior into valid, state-consistent network traces through a compiler backing.

Jane: It really shows how much better things get when you separate the learning part from the deterministic realization part of a system.

Lu: I think it gives sequence models a usable interface—that latent space—so they can operate on something structured rather than just raw header fields, which is a big deal for generative AI.

Meng: From an engineering standpoint, this means we can build tools that generate traffic that actually respects protocol constraints from the start instead of relying on messy repairs later.

Lalam: I see this as a cultural shift where AI systems become less about pattern matching raw data and more about understanding and respecting the underlying structure of protocols.

Tom: They’ve shown impressive results on CICIDS2017 Monday, matching flow counts within zero point zero three percent, which is solid evidence that this method works for high-fidelity traffic synthesis.

Jane: It’s a big step because it solves that fundamental problem where learned choices get tangled up with rigid protocol rules.

Lu: The future work they hint at is using those latent prefixes and suffixes to train conditional flow-matching backbones, which lets models test their ability to produce specific sequences of actions.

Meng: That suggests a way for AI to actually learn the *process* of generating a trace, not just the final output.

Tom: So we’re moving from just getting good statistical summaries to building tools that create high-fidelity, state-aware network traces using this TraceCodec approach.

Jane: It’s about making sure that what an AI generates looks like something a real network could actually produce, which is crucial for security testing and validation.

Lu: This separation of concerns between the behavioral codec and the deterministic compiler opens up new avenues for modeling complex, stateful traffic patterns.

Meng: I think this means we can focus our engineering efforts on building these state-aware decoding layers instead of just trying to make raw field decoders work better.

Junhui Ding, Xinchen Zhang, Xiaohui Xie, Shinan Liu

Tsinghua University · University of Hong Kong

cs.NI, cs.LG

Submitted: 2026-05-28

Updated: 2026-10-11

Code: https://github.com/appneta/tcpreplay

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 92/100

The gist: The gist The core problem is the decode interface, which mixes behavioral choices with protocol consequences, forcing packet realization to depend on post-hoc heuristic repair TRACECODEC Overview

Key concepts

Packet Action
A structured record representing the behavioral choices an endpoint makes at a single packet step. It captures protocol details like direction and control events alongside payload behavior and timing, providing a clean input for the learning model.
Latent Codec
The learned component that maps structured packet actions, local context, and timing values into a continuous latent vector. This vector acts as an intermediate representation that is easier for downstream sequence models to process before final packet realization.
Deterministic Compiler
A compiler that takes the decoded packet actions and explicitly manages deterministic state evolution. It resolves issues like flow template assignment, transport state changes (e.g., TCP sequence numbers), and legality constraints to produce concrete, valid packets.
State-Aware Decoding
The process where the decoder predicts detailed packet fields conditioned on the latent code, runtime context, and coarse decoded fields. This ensures that the resulting actions are not just locally accurate but also consistent with the necessary transport state required for correct packet rendering.

Terminology

Summary

The gist The core problem is the decode interface, which mixes behavioral choices with protocol consequences, forcing packet realization to depend on post-hoc heuristic repair

TRACECODEC Overview

TRACECODEC is a compiler-backed neural codec that decouples learned packet behavior from deterministic packet realization It has three layers: first, packet parsing lifts each packet into a structured packet action, a local flow context summarizing recent flow-local history, and an interpacket timing value Second, the learned codec maps each packet-action record into a continuous latent vector and decodes that vector back into timed actions before deterministic state realization emits the final PCAP Third, the deterministic compiler compiles decoded packet actions into concrete packets under explicit flow state, resolving flow-template assignment, transport-state evolution, legality constraints, and packet rendering

Packet Actions and Latent Codec

A packet action is a structured record of the behavioral choices an endpoint makes at one packet step It includes protocol, direction, control event, payload behavior, and timing The learned component is a contextconditioned codec over packet-action records It maps each packet action, local context, and timing value into a continuous latent vector and decodes that vector back into packet-action fields suitable for deterministic compilation The decoder is staged, first predicting coarse packet-action semantics such as flow token, flow event, protocol, IP family, direction, and TCP control event It then predicts detail fields conditioned on the latent code, runtime context, and decoded coarse fields The learned interface exposes state-aware packet actions in a form better matched to downstream sequence models

Deterministic Compiler and Realization

The deterministic compiler produces pˆ = C(aˆ1:T, ∆ct1:T) given decoded actions and timing values The compiler first resolves the flow slot encoded by flow token, mapping it to a synthetic endpoint template via a FlowTable Direction determines whether the packet uses the template endpoints directly or swaps them For TCP traffic, the compiler maintains per-flow, per-direction transport state It assigns deterministic initial sequence numbers and tracks relative sequence and acknowledgment progress after events like SYN, SYN-ACK, FIN, RST, DATA, and ACK-like events The compiler performs constrained lowering to preserve legality by construction

Evaluation and Results

TRACECODEC reproduces packet count, inter-arrival time, and flow count within 1% of the real trace on CICIDS2017 Monday Structural diagnostics show that TRACECODEC preserves TCP state transitions and multi-flow interleaving that raw-field decoders fragment The oracle shows that the packet-action representation and deterministic compilation form an almost lossless symbolic path, with zero packet-count, protocol, and flow-count error on CICIDS2017 Monday TRACECODEC stays close to this path on packet and flow structure, with 0.00% count error, 0.03% flow-count error, and 9.23% session-duration distortion The state-aware compiler makes the transition structure a consequence of decoded actions under explicit transport state, rather than a set of independently decoded raw fields that must agree after the fact

Latent Interface Usefulness

TRACECODEC is intended as a codec layer for packet-trace generators, and its latent space is a usable interface for an imperfect downstream generator The latent-interface experiment asks whether a generator can stay on a usable decode surface while matching the actual continuation trace, rather than looking good only relative to a representation-specific oracle decode TRACECODEC again provides the most reliable decode surface, with nearly all suffixes usable, full median valid-packet coverage, and substantially lower coverage-adjusted timing, packet-size, flow-size, and session-duration distances than the raw-field latent controls The ablation study shows that the full model reconstructs packet actions at 99.73% and keeps decoded count, IAT, and flow-count errors below 1%, giving a reference point for the learned interface Removing fine timing raises IAT distortion from 0.59% to 13.37% while leaving other metrics close, showing that packet timing must remain explicit Removing TCP state cues even improves action accuracy to 99.95%, but raises TCP event-profile distance to 138.85 percentage-points, showing that local reconstruction accuracy does not guarantee transport-correct decoded traces TRACECODEC suggests a practical path toward higher-fidelity traffic generators built around explicit state-aware decoding rather than heuristic packet repair

Privacy Considerations

TRACECODEC keeps the variables needed for packet-/transport-level decode, such as timing, flow-slot identity, protocol, direction, control semantics, bounded acknowledgment advancement, receivewindow structure, and payload-length behavior In particular, the learned representation omits raw IP addresses and application payload bytes The practical implication is that the training representation is not a lightly disguised copy of the original packet headers but a structured behavioral representation whose purpose is to preserve packet-and transport-level semantics while suppressing directly identifying fields The implementation also supports randomized per-run flow salts so that the same latent code can be decoded into different synthetic endpoint assignments across runs, improving unlinkability between decoded traces and any specific original address assignment

Conclusion

TRACECODEC embodies this separation through a packet-action interface and compiler-backed decoding, which turns learned outputs into valid packet traces under explicit transport state This approach avoids the brittle failure modes of raw-field reconstruction by preserving the structure needed for packet-level generation and making the remaining errors easier to locate, especially in TCP dynamics and long-tail timing While TRACECODEC still can not support application layer protocols, it suggests a practical path toward higher-fidelity traffic generators built around explicit state-aware decoding rather than heuristic packet repair

--- Page 1 ---

TRACECODEC: A Compiler-Backed Neural Codec for Stateful Multi-Flow Network Traffic Traces Junhui Ding Tsinghua University Xinchen Zhang University of Hong Kong Xiaohui Xie Tsinghua University Shinan Liu University of Hong Kong Abstract Critical networking workflows require high-fidelity packet captures (PCAPs) for testing, security analysis, and protocol validation, not just statistical flow-level summaries Recent packet generators have demonstrated protocol-constrained PCAP synthesis, but they universally decode directly to raw packet fields That interface entangles learned behavioral choices with deterministic protocol consequences, which forces packet realization to depend on post-hoc heuristic repair We identify this decode interface as the fundamental bottleneck and present TRACECODEC, a state-aware neural codec for stateful multi-flow traces TRACECODEC lifts each packet into a timed packet action with explicit flow slots and transport cues, then learns a continuous per-packet latent A deterministic compiler lowers decoded actions back to PCAPs, owning endpoint assignment, TCP state, legality constraints, and packet rendering The latent layer exposes a generator-facing sequence space, so downstream traffic models can operate on packet-action latents rather than raw header fields On CICIDS2017 Monday, TRACECODEC matches packet count, protocol composition, and flow population to within 0.03% Raw-field baselines under the same non-repair policy distort flow counts and TCP state by orders of magnitude Structural diagnostics show that TRACECODEC preserves TCP state transitions and multi-flow interleaving that raw-field decoders fragment This work establishes a new foundation for high-fidelity packet-trace generation

--- Page 2 ---

Raw-field decode behavior + protocol consequences entangled PCAP Flow A Flow B Flow C · · · raw fields dt flow dir flags seq ack len cksum · · · · · · = behavior protocol timing flow dir event seq len cksum legal raw-field model generated rows flow dir flags seq ack ambiguous fix A B state drift SYN DATA × trace distortion TraceCodec behavior learned, realization compiled PCAP Flow A Flow B Flow C packet actions dt slot dir event latent z1 z2 z3 z4 learned behavior timing slot dir event clean contract state-aware compiler Flow slots TCP automaton S E D seq/ack SYN DATA

--- Page 3 ---

Figure 1: Comparison of decode interfaces. Raw-field decode entangles behavior with protocol consequences, making generated rows repair-dependent TraceCodec learns packet actions above a clean contract and compiles them into state-consistent packets

--- Page 4 ---

The core problem is the decode interface Raw packet fields mix two kinds of variables: behavioral choices that a model should learn and protocol consequences that packet state should determine When a model emits both on the same surface, final PCAP realization depends on repair rules rather than on a clean decode contract This problem appears in three properties that make packet traces usable (1) Timing is an important traffic behavior.

Improvements for AI systems

  1. textbf Learned Interface for Generative Models: TRACECODEC Latent Space Enables Sequence Modeling . This allows downstream traffic models to operate on latent packet-action sequences rather than raw header fields, exposing a generator-facing sequence space.

  2. textbf High-Fidelity Packet Generation: Deterministic Compilation Guarantees Protocol Legality . The system utilizes a deterministic compiler that lowers decoded actions back to PCAPs, owning endpoint assignment, TCP state, legality constraints, and packet rendering, ensuring the output is state-consistent and not subject to post-hoc heuristic repair.

  3. textbf Enhanced Traffic Semantics: Disentangling Behavior from Constraints . By identifying the decode interface as the fundamental bottleneck and mapping it to timed packet actions, AI systems can learn representations that capture realistic timing behavior while separating it from deterministic protocol consequences, which is critical for generating scenarios that are protocol-constrained.

  4. textbf Robustness Against State Drift: State-Aware Compiler Preserves TCP Transitions . The compiler maintains per-flow, per-direction transport state and updates state after SYN, SYN-ACK, FIN, RST, DATA, and ACK-like events, which prevents the trace distortion seen in raw-field baselines by ensuring that state transitions are preserved.

  5. textbf Efficient Trace Synthesis: Latent Interface Training for Sequence Matching . By training a conditional flow-matching backbone on each representation using latent prefixes and suffixes, AI systems can be tested on whether they can produce sampled suffix latents decoded back through the corresponding representation and compared against real held-out suffix PCAPs.

  6. textbf Privacy-Aware Traffic Analysis: Suppression of Direct Identifiers . TRACECODEC suppresses raw IP addresses, raw transport ports, MAC addresses by re-instantiating synthetic endpoints from flow-slot identity, allowing for analysis while reducing the direct exposure of identifying information in generated traces.

Abstract

Critical networking workflows require high-fidelity packet captures (PCAPs) for testing, security analysis, and protocol validation, not just statistical flow-level summaries. Recent packet generators have demonstrated protocol-constrained PCAP synthesis, but many operate on raw or near-raw packet representations. These representations can entangle learned behavioral choices with deterministic protocol rules and make packet correctness depend on post-hoc repair. We identify the decode interface as a key bottleneck and present TraceCodec, a compiler-backed neural codec for stateful multi-flow traces. TraceCodec lifts each parsed packet into a timed packet action with explicit flow slots and transport cues, learns a continuous per-packet latent, and uses a state-aware deterministic compiler to decode actions to PCAPs while managing endpoint assignment, TCP sequence and acknowledgment state, declared compatibility constraints, and packet rendering. On CICIDS2017 Monday, TraceCodec matches packet count, protocol composition, and flow population to within 0.03%; raw-field baselines produce substantially larger flow-count and TCP-event errors under the same non-repair policy. In a matched flow-matching continuation setup, TraceCodec yields complete decoded suffixes in 100% of CICIDS2017 cases and 98.44% of MAWI cases, versus at most 10.94% for raw-field latent controls. Structural diagnostics further show that TraceCodec preserves TCP transitions and multi-flow interleaving. Together, these results support TraceCodec as a generator-facing latent interface with a deterministic path back to usable packet traces.

Sources

Related papers