Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding

The gist The Elastic Time method introduces a dynamic frame-rate bottleneck that converts fixed-frame-rate autoencoders into dynamic ones by learning a lightweight latent predictor to decide which frames can be skipped during inference, enabling efficient rate control.

Proposed Method

1. The core technical contribution is a lightweight predictor trained to predict future latent frames from past context.

2. This predictor models short-term latent dynamics along the temporal axis by predicting the next frame given the previous predicted frame and recurrent state.

3. The process involves a chunk procedure that partitions the latent sequence into N variable-length segments according to a predictability criterion, retaining one “anchor” frame from each segment.

4. The dechunking procedure produces the full-length reconstructed latent sequence yˆ, using P.

5. Within each segment m (with length lm > 1), we perform one autoregressive predictor rollout for lm − 1 steps, starting from the anchor frame yam 1), we perform one autoregressive predictor rollout for lm − 1 steps, starting from the anchor frame yam>.

Boundary Selection

The chunking is guided by the assumption that latent predictability implies redundancy: if a segment of latents can be approximated from its anchor via P, then these frames can be dropped.

Two solvers are proposed for boundary selection: (i) an approximate but efficient greedy procedure and (ii) an exact dynamic program (DP)

The greedy algorithm runs in O(T log T) time after initializing local costs.

The DP formulation defines the cost as c(s, l) = Xl−1 i=1 d yˆ (s) s+i, ys+i.

Training Objectives

The model is trained with two coupled objectives: (i) accurate multi-step latent prediction and (ii) preservation of reconstruction quality after chunking/dechunking.

The predictor P is trained using a rollout prediction loss Lroll which penalizes multi-step rollout error.

The model also uses an auxiliary prediction loss Lvalid computed on the frames replaced during this transformation.

Evaluation and Results

The evaluation compares four main variants: two greedybased models, one trained at a fixed kept fraction ρ = 0.5 (denoted ET-greedy@0.5) and one trained over a range ρ ∼ U(0.5, 0.995) (ET-greedy), and two DP-based models.

The results show that greedy and DP chunking perform similarly.

ET generally achieves lower mel-d and FAD on SongDescriber and DAPS, with the clearest gains in FAD.

The conclusion is that dynamic frame-rate scalability could improve the efficiency of downstream generation and long-context audio modeling.

Conclusion

Elastic Time enables framerate scalable models via content-adaptive latent decimation.

The method is implemented as a Re-Bottleneck plug-in on top of a frozen pretrained autoencoder.

It achieves competitive efficiency-quality tradeoffs relative to fixed-rate and prior dynamic chunking baselines.

The overall impact is that the method allows for deployment-time rate control without external semantic supervision.

Acknowledgments

This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government.

References

The references listed are [1] A. Defossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023. [5] Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” in Proc ICASSP IEEE 2025 1–5 <ref:2606.27320#pg6>. [17] Y. Chae, W. Choi, Y. Takida, J. Koo, Y. Ikemiya, Z. Zhong, K. W Cheuk et al., “Variable bitrate residual vector quantization for audio coding,” in Proc ICASSP IEEE 2025 1–5. [19] J. Li, Y. Qian, Y Hu, leying zhang et al., “Flexicodec: A dynamic neural audio codec for low frame rates,” in Proc ICLR 2026. [23] D. Bralios, J. Casebeer, and P. Smaragdis, “Re-bottleneck: Latent restructuring for neural audio autoencoders,” in 2025 IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP) IEEE 2025 1–6 <ref:2606.27320#pg2>. [43] C. J. Steinmetz and J. D. Reiss, “auraloss: Audio focused loss functions in pytorch,” in Digital music research network one-day workshop (DMRN+ 15) 2020 <ref:2606.

1. The Elastic Time (ET) module enables deployment-time rate control by learning a lightweight latent predictor used to decide which frames can be skipped and later reconstructed. This allows downstream models to adapt their temporal resolution dynamically based on the required compression factor, facilitating more efficient deployment.

2. The system supports frame-rate scalability by allowing the model to adjust its effective latent sequence length at inference time. By using a predictability criterion guided by the learned predictor P, it can perform efficient greedy boundary selection or an exact dynamic programming procedure to select segments, effectively converting fixed-frame-rate autoencoders into dynamic ones.

3. The training objective optimizes for both prediction accuracy and reconstruction quality simultaneously. The loss function is defined as Lpred = λrollLroll + λvalidLvalid, ensuring that the predictor P is trained to produce stable rollouts while maintaining compatibility with the base model's reconstruction ability, which is crucial for preserving perceptual quality during dynamic decimation.

4. The Re-Bottleneck framework allows for the reuse of pretrained, publicly available fixed-rate models by focusing learning on a latent-space module that specializes in dynamic temporal allocation while preserving the base model’s strong reconstruction ability. This significantly reduces the computational cost associated with training audio autoencoders from scratch.

5. The method provides a flexible mechanism for adjusting temporal resolution, which is expected to facilitate more efficient downstream modeling for generation and long-context tasks, such as autoregressive next-token prediction or diffusion-based generative models that are bottlenecked by latent sequence length.

Neural audio autoencoders have become a core component of compression, feature extraction, and generation. However, while existing systems support variable bitrate, the vast majority of models still operate at a fixed latent frame-rate, allocating equal temporal budget to regions with very different information density, which can result in unnecessarily long sequences. We introduce Elastic Time, a dynamic frame-rate bottleneck that converts fixed-frame-rate autoencoders to dynamic ones. Our method learns a lightweight latent predictor used to decide which frames can be skipped and later reconstructed, enabling efficient greedy boundary selection at inference. Experiments show our method enables deployment-time rate control while improving efficiency-quality tradeoffs relative to baselines. Overall, we provide a flexible mechanism for adjusting temporal resolution in audio autoencoders, potentially facilitating more efficient downstream modeling for generation and long-context tasks.