HART: Hybrid Attention-RWKV Time series Foundation Model

arXiv:2610.12883 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "HART: Hybrid Attention-RWKV Time series Foundation Model".

Jane: The gist: HART, a Hybrid Attention-RWKV Time-series Foundation Model, improves over pure RWKV and Transformer on medium and long horizons while maintaining comparable short-horizon performance,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're talking about this new paper called HART: Hybrid Attention-RWKV Time series Foundation Model. The big idea here is to make time series foundation models able to predict an entire future horizon in just one forward pass using non-autoregressive multi-step forecasting or NAF.

Jane: That sounds like a lot of prediction at once, Tom. So, what's the main problem this paper is trying to fix with this HART model?

Tom: Well, they found that RWKV-seven struggles with long horizons in these NAF experiments <ref:2610.12883#pg1>. When you ask it to predict future steps without any new observations, it starts degrading pretty fast.

Lu: The core issue they’re looking at is how the recurrent state evolves across that prediction horizon when you feed it missing inputs, which requires the model to keep making transitions without new data.

Meng: If we're talking about deployment, what does this degradation look like for us? Does it just mean bad forecasts later on?

Tom: It suggests a mechanism they call Placeholder Idling Decay or PID. Essentially, they found that repeated state transitions can weaken the historical information when the placeholder-driven writes don't compensate enough.

Jane: Placeholder Idling Decay sounds complicated. Can you tell us more about what that means for how the model remembers things?

Lu: Their analysis of RWKV-seven’s generalized delta rule points to this mechanism because they see reduced write norms alongside similar transition norms, which is consistent with PID <ref:2610.12883#pg1,analysis of RWKV-7’s generalized delta rule>.

Tom: And layer-wise analysis showed that state-injection norms were smaller in most layers, while the transition-matrix norms stayed pretty similar between the historical and prediction regions, which supports that PID idea.

Meng: That sounds like a stability issue we worry about when we scale things up to production. So what's their proposed solution then?

Jane: The paper proposes HART, which is this Hybrid Attention-RWKV Time series Foundation Model that alternates between attention and RWKV blocks.

Tom: It combines direct access to historical tokens with recurrent temporal modeling. They use this alternating structure because the RWKV blocks handle temporal compression in the historical region, while the Transformer blocks retrieve information in the prediction region through global attention.

Lu: This ordering allows each RWKV block to have attention-conditioned representations at prediction positions, meaning its writes can depend on retrieved history even when the original future values are masked.

Meng: So it's trying to get the best of both worlds: having the speed of recurrent models and the long-range context of attention. What does this look like in practice?

Tom: They trained HART using a contiguous patch masking method with next-patch prediction, which exposes it to NAF inference mode during training. The model outputs nine equidistant quantiles and uses a Pinball loss for both the CPM and autoregressive tasks.

Jane: So how does this hybrid architecture compare to the pure RWKV model they were testing against? Are there any hard numbers showing where HART wins?

Paper summary: Tom: Controlled experiments on GIFT-Eval and TIME-Benchmark show that HART significantly improves over pure RWKV at medium and long horizons while keeping comparable short-horizon performance.

Meng: And what about the aggregate performance when you compare it to a standard pure Transformer model of a similar size? Does HART beat that too?

Tom: Yes, it achieves better aggregate performance than a comparably sized pure Transformer. With three sixty million parameters, HART gets an MASE comparable to Toto-two point zero-1B on GIFT-Eval with zero point seven five two versus zero point seven five four, using only about one twenty-eighth of its parameters for that score.

Jane: That's a lot of context being packed into a model that uses much less than a pure Transformer to get those results. So what does this mean if we're just looking at the practical impact on forecasting?

Tom: It shows HART is competitive with other top models, like Toto-two point zero-1B, on long horizons while staying very close to the performance of simpler models on short horizons <ref:2610.12883#pg1>.

Lu: The synthetic experiments are interesting because RWKV actually outperforms both HART and Transformer under NAF on SE4, but HART performs best when tested on SE3.

Meng: That variation suggests that hybrid architectures aren't universally superior; their effectiveness seems signal-dependent, depending on the type of time series data you're dealing with.

Jane: So, while this paper focuses heavily on the RWKV-seven case to explain PID, what are they saying about the broader field of time series foundation models <ref:2610.12883#pg1>?

Tom: The analysis is restricted to RWKV-seven and it's important to remember that the norm diagnostics don't directly measure how much historical information is actually retained <ref:2610.12883#pg1>.

Lu: And they also noted that their comparison with teacher-forced autoregressive models doesn't fully isolate the PID mechanism from other architectural effects.

Meng: For deployment, if we use this HART model, what are we looking at in terms of maintenance? Is it more complex to maintain than a standard Transformer?

Tom: It's a hybrid, so it adds complexity on the architecture side because you're managing two different modeling components interacting.

Jane: So to wrap up this look at HART: Hybrid Attention-RWKV Time series Foundation Model, the authors are showing how combining attention and RWKV can mitigate that decay issue they called PID.

Tom: They prove it by demonstrating improvement over pure RWKV on medium and long horizons while keeping short horizons stable, and they show it beats a comparable Transformer on aggregate metrics.

Lu: The implication is that alternating between recurrent blocks for local modeling and attention for global retrieval is a viable way to handle the trade-off between temporal memory and long-range context in forecasting models.

Meng: For us shipping models, it means we might have a new architectural pattern to test when we run into those long-horizon prediction problems that pure recurrence struggles with.

Jane: It gives us a concrete model, HART, that shows how you can combine different AI components to get better performance across different forecasting needs.

Conclusion: Tom: So we're wrapping up this look at HART, the Hybrid Attention-RWKV Time series Foundation Model.

Jane: It’s basically this new way of building a model that mixes attention and recurrent blocks to handle long-term forecasting better than just using one type of architecture alone.

Tom: The authors are Taha Aksu and Gerald Woo, and they focused on a specific problem in non-autoregressive multi-step forecasting, or NAF.

Jane: They found that the standard RWKV model gets pretty shaky when you ask it to predict way out into the future without any new data points.

Tom: That shakiness leads them to this idea called Placeholder Idling Decay, or PID—basically, how repeated state changes can weaken historical memory when those placeholder writes aren't enough compensation.

Jane: The authors propose HART as the fix by alternating between attention blocks and RWKV blocks, so the recurrent part handles local history while the attention part reaches back globally.

Tom: And they show that this hybrid setup beats pure RWKV on medium and long horizons, while still keeping up well on shorter prediction tasks.

Jane: It’s interesting because they also beat a similarly sized pure Transformer model on aggregate metrics when looking at those longer predictions.

Tom: The implication is that for time series foundation models, mixing different types of memory—like recurrent and attention-based—might be the way to go for better performance across different prediction lengths.

Jane: It’s definitely something to think about if you’re trying to build a model that needs both short-term context and long-range planning.

Tom: But they also pointed out that this mechanism is specific to the RWKV-seven setup, so we need to be careful when applying it to other architectures.

Jane: Exactly, so next time we look at a new paper on forecasting models, we should always check if they’re talking about the same type of model and the same kind of prediction task.

Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai

School of Computer Science, Guangdong University of Technology · Peng Cheng Laboratory · College of Science, Shantou University

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist: HART, a Hybrid Attention-RWKV Time-series Foundation Model, improves over pure RWKV and Transformer on medium and long horizons while maintaining comparable short-horizon performance,

Key concepts

Non-autoregressive multi-step forecasting (NAF): This is a task where the model must predict an entire future time series at once, without generating one step after another. In this setup, future data points are treated as missing inputs, forcing the model's internal state to evolve through prediction without new real observations.
Placeholder Idling Decay (PID): This mechanism describes how RWKV-7 suffers in NAF tasks. It suggests that when the model uses placeholder values for missing future data, repeated state updates can weaken its ability to retain and use important historical information, especially if the placeholder writes don't provide enough useful compensation.
HART Architecture: HART alternates between Transformer blocks and RWKV blocks. The RWKV blocks handle local temporal modeling of past data, while the Transformer blocks use global attention to access historical tokens during prediction. This hybrid design aims to bypass the decay issues seen in pure recurrent models.
Contiguous Patch Masking (CPM): This is a training technique used for NAF tasks. During training, the model is exposed to forecasting by masking future patches and training it to predict the next patch based on its current context. This simulates the inference mode where only partial future information is available.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes

Summary

The gist: HART, a Hybrid Attention-RWKV Time-series Foundation Model, improves over pure RWKV and Transformer on medium and long horizons while maintaining comparable short-horizon performance, achieving better aggregate performance than comparably sized pure Transformer

How it works

HART is a Hybrid Attention-RWKV Time-series Foundation Model that combines direct access to historical tokens with recurrent temporal modeling by alternating attention and RWKV blocks Layer-wise analysis reveals reduced state-injection norms in most layers, while transition-matrix norms remain similar between historical and prediction regions, consistent with Placeholder Idling Decay (PID) The architecture consists of an alternately stacked backbone where each group consists of one Transformer block followed by one RWKV block, giving 2L layers in total This order gives each RWKV block attention-conditioned representations at prediction positions, allowing its writes to depend on retrieved history even when the original future values are masked

Placeholder Idling Decay (PID)

The paper characterizes Placeholder Idling Decay (PID) in RWKV-7 under NAF through generalized delta-rule analysis and layer-wise norm measurements PID suggests that repeated state transitions can attenuate historical information when placeholder-driven writes provide insufficient compensation Analysis of RWKV-7's generalized delta rule suggests this mechanism, where reduced write norms alongside similar transition norms are consistent with PID The analysis restricted to RWKV-7 shows that the prediction-to-history ratio of Kt falls between 0.60 and 0.87 in the first seven layers Layer 12 exhibits a reversal, with a ratio of 1.34, highlighting variation across depth

HART Architecture and Training

HART proposes that attention bypasses the recurrent decay path by alternately stacking RWKV and Transformer blocks RWKV blocks perform temporal compression and local modeling in the historical region, while Transformer blocks retrieve historical information in the prediction region through global attention, bypassing the stepwise decay path The training objective adopts contiguous patch masking (CPM) with next-patch prediction, exposing the model to NAF inference mode during training The model outputs Qquantile = 9 equidistant quantiles and uses the Pinball loss to train on both CPM and autoregressive tasks

Experimental Results

Controlled experiments on GIFT-Eval and TIME-Benchmark show that HART significantly improves over pure RWKV at medium and long horizons while maintaining comparable short-horizon performance On aggregate metrics, HART reduces MASE relative to the pure Transformer on both benchmarks The synthetic experiments show that RWKV outperforms both HART and Transformer under NAF on SE4, whereas HART performs best on SE3 This variation supports examining hybrid architectures across diverse signal regimes rather than expecting universal dominance

Limitations and Future Work

The mechanistic analysis is restricted to RWKV-7, and the norm diagnostics do not directly measure retained historical information The comparison with teacher-forced AR does not isolate PID from all other architectural effects The paper suggests that while HART improves over RWKV on Medium and Long horizons, the benefit of the hybrid depends on the evaluated tasks, as seen in synthetic experiments The sensitivity to missing future observations is signal-dependent, with RWKV exhibiting the largest NAF–AR gap for certain signals like SE3

AI Use Statement

The authors used DeepSeek-V4 Pro to assist with writing, polishing, and translation of this manuscript They also used OpenAI Codex for rhetorical editing, consistency checks, and revision suggestions The authors take full responsibility for the content and final version

References

Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393

Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, et al. Chronos-2: From univariate to universal forecasting. arXiv preprint arXiv:2510.15821

Rafiq Asghar, Francesco Riganti Fulginei, Michele Quercio, and Assia Mahrouch. Artificial neural networks for photovoltaic power forecasting: A review of five promising models. IEEe Access, 12: 90461–90485

Andreas Auer, Patrick Podest, Daniel Klotz, Sebastian Böck, Günter Klambauer, and Sepp Hochreiter. Tirex: Zero-shot forecasting across long and short horizons with enhanced in-context learning. Advances in Neural Information Processing Systems, 38:57529–57580

Jianrong Cao, Xing Sheng, and Zongtao Duan. Mstf: A multi-scale long-term spatio-temporal traffic flow forecasting framework. IEEE Internet of Things Journal, 2026

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688

Kun Feng, Shaocheng Lan, Yuchen Fang, Wenchao He, Lintao Ma, Xingyu Lu, and Kan Ren. Kairos: Towards adaptive and generalizable time series foundation models. arXiv e-prints, pp. arXiv–2509

Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. From tables to time: Extending tabpfn-v2 to time series forecasting. arXiv preprint arXiv:2501.02945

Muaaz Bin Kaleem, Heng Li, Zeyu Zhu, Xiaolong Chen, Yunsheng Fan, Lisen Yan, and Weirong Liu. Lag-llama-based remaining useful life prediction for lithium-ion batteries of electric vehicles with auto-correlation analysis. IEEE Transactions on Transportation Electrification, 2026

Emaad Khwaja, Chris Lettieri, Gerald Woo, Eden Belouadah, Marc Cenac, Guillaume Jarry, Enguerrand Paquin, Xunyi Zhao, Viktoriya Zhukov, Othmane Abou-Amal, et al. Toto 2.0: Time series forecasting enters the scaling era. arXiv preprint arXiv:2605.20119

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International conference on learning representations, 2021

Maurice Kraus, Felix Divo, Devendra Singh Dhami, and Kristian Kersting. xlstm-mixer: Multivariate time series forecasting by mixing via scalar memories. Advances in Neural Information Processing Systems, 38:6547–6581

Jinho Lee and Jincheol Kim. Fast and effective prediction of spatiotemporal nonlinear dynamics using the tailored rwkv architecture.

Improvements for AI systems

Bold header: Hybrid Attention-RWKV Temporal Modeling

This improvement combines direct access to historical tokens with recurrent temporal modeling by alternating attention and RWKV blocks, allowing for both local modeling and global access. This allows the system to leverage recurrent temporal modeling in the historical region while using attention to bypass the recurrent decay path.

Bold header: Mechanism-Driven Degradation Analysis

By characterizing Placeholder Idling Decay (PID), the system can identify when repeated state transitions can attenuate historical information when placeholder-driven writes provide insufficient compensation. This allows for targeted architectural modifications to prevent information loss during long-horizon NAF prediction.

Bold header: Enhanced Long-Horizon Forecasting Accuracy

The resulting HART model achieves better long-horizon and overall MASE than comparably sized baselines on GIFT-Eval and TIME-Benchmark, specifically showing improvements over pure RWKV on Medium and Long horizons.