OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning

arXiv:2609.40265 · cs.LG · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning".

Jane: The gist: OpenTSLM TeeMoE introduces a generalist time-series language model that unifies numerical forecasting, context-conditioned prediction, and temporal reasoning by combining independently trained low-rank experts within a shared backbone.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now we’re moving into the specifics of what OpenTSLM TeeMoE actually is, which is titled "OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning."

Jane: This model aims to solve a big problem: time series foundation models are usually split. Numerical specialists are good at forecasting numbers, but language models are good at context and analysis.

Lu: The paper introduces OpenTSLM TeeMoE as a generalist time-series language model that can forecast directly from observed time series, reason over textual context and temporal patterns.

Meng: Can you break down what that means in simpler terms for someone who doesn't read the technical details?

Tom: Think of it this way: instead of needing three separate models—one just for numbers, one just for text understanding, and one just for reasoning—you have one model that can do all three things.

Jane: So, when you give it a time series and a question about it, it doesn't have to switch between different tools; the model figures out which tool to use internally.

Lu: It does this by using that mixture of experts setup we talked about earlier, where a controller decides which expert gets to handle the request.

Meng: So the controller is basically making a decision on the fly about which expert weights to use? That sounds like it could be slow if it has too many choices.

Tom: The controller uses that representation of the request to assign weights to the three experts and choose between two possible outputs: a numerical decoder or a language-model head.

Jane: And that means the model is intelligently routing the input based on what’s in the text, deciding whether it needs a forecast number or an explanation.

Lu: It’s about unifying these heterogeneous capabilities without diluting the performance of strong task-specific models, which is a central question they are addressing in this work.

Meng: And they show that by doing it this way with LoRA library composition, you can largely retain specialist performance across different training tasks.

Tom: That’s what makes it interesting; broad task coverage doesn't necessarily hurt the performance of the individual experts when they're composed in this specific manner.

Jane: So, the main point is that we can get a model that handles forecasting, context prediction, and reasoning all at once, and it does so without completely sacrificing what those specialists are good at.

Lu: The structure of OpenTSLM TeeMoE is built from independently trained experts within a shared backbone.

Meng: So the backbone is one thing everyone uses, but the specialized knowledge lives in these separate adapters?

Tom: Precisely. The backbone is frozen, and only the three adapters are trained separately for their specific tasks.

The paper's summary: Jane: Now let’s look at how they actually built this thing, because the summary of OpenTSLM TeeMoE talks about a staged training approach to fit these components sequentially.

Lu: They tackle this in several stages. First, they have aggregation training where they fit an XGBoost regressor to combine eight selected forecasting models and learn a constant weight that blends its forecast with Toto-FnF.

Tom: That aggregation expert learns a bounded correction from the reference median to the observed future using a weighted SmoothL1 loss and a squared-edit penalty, which is what refines the forecasts.

Meng: It sounds like they are using an ensemble of eight models just to get one better reference forecast before refining it further?

Tom: That’s right. And then there’s native forecasting training, where they train for one epoch on twenty thousand contextual forecasting examples using teacher-forced cross-entropy with forward KL regularization.

Jane: So the second stage focuses on the native forecasting expert learning to generate those predictions directly from history and text based on those examples.

Lu: Next is temporal analysis training, where the temporal analysis expert learns to answer analytical questions about observed signals by using response-token cross-entropy and forward and reverse KL regularization.

Meng: So one part focuses on answering questions about the data itself, rather than just making a prediction? Like pattern recognition?

Tom: Exactly. And finally, they have controller fitting, where they train the controller to minimize a loss function that combines prediction loss for each capability and an output-format loss.

Jane: So the whole training process is sequential: first build the components, then train them one by one, and finally tune the controller to manage how they all interact during a request.

Lu: The key contribution here is using this staged approach to study how mixture of experts composition affects TSLM capabilities and training costs.

Meng: And they provide comparisons of specialists versus composition versus joint training to guide future generalist time-series language models.

The paper's improvements: Tom: When we talk about improvements, the paper points out a few key things that are good additions to the original idea.

Jane: One improvement is showing how they use LoRA-library composition to demonstrate capability preservation, which is a big thing because it proves those specialist strengths stick when you combine them.

Lu: They show that this compositional formula for an adapted matrix in layer l is W'l = Wl + sum e=one pi e ∆We,l where ∆We,l is expert e’s low-rank update <ref:2609.40265#pg1>.

Meng: That means the experts keep their own low-rank updates even when they are mixed into the shared backbone? It keeps things modular at a granular level.

Tom: Right. And they also mention how they handle training costs in Table twelve comparing the modular cost of thirty point seven five six H100 GPU-hours against the joint recipe cost of fifty-eight point six five seven H100 GPU-hours for joint training on eight GPUs.

Jane: So, the practical improvement here is that separate specialization costs less time and money when you want to build specific capabilities, which is a huge win for iterative development.

Lu: The comparison principle they use is about retaining the specialists’ data exposure and prediction interfaces while replacing separate adapters with shared parameters in joint training.

Meng: So the modular design allows you to keep the experts’ specific recipes intact while still being able to retrain just one expert without retraining all three.

Conclusion: Tom: So, let’s wrap up OpenTSLM TeeMoE by summarizing its main points for listeners.

Jane: In short, this paper introduces a generalist time-series language model that brings numerical forecasting, context-conditioned prediction, and temporal reasoning into one system using a mixture of experts structure.

Lu: The implication is that you can achieve competitive performance across benchmarks where specialized models usually dominate with this unified approach.

Meng: It proves that broad task coverage can coexist with strong predictive performance when structured correctly through modular composition.

Tom: They’ve shown a practical route to generalist time-series modeling by training each capability with the objectives and resources it needs, then bringing them together without giving up their strengths or the flexibility to develop them independently.

Jane: It’s a strong piece of work because it demonstrates how we can get predictive power across different domains in this new unified model.

Lu: We're excited to see what researchers do next with these findings on composition and training costs as they push the boundaries of these models.

Tony Chen, Timo Stoffregen, Maxwell Xu, Thomas Kaar, Martin Maritsch, Geremia Pompei, Nicolas Zumarraga, Robert Jakob, †Paul Schmiedmayer†Patrick Langer†Juncheng Liu

Columbia University · Stanford University · ᵉionic Labs Switzerland · Google Agentic Systems Lab ETH Zurich Switzerland · University of Pisa Italy · National University of Singapore

cs.LG

Submitted: 2026-09-30

Updated: 2026-10-07

Comments: 39 pages, 2 figures. Code: https://github.com/OpenTSLM/OpenTSLM-TeeMoE ; model: https://huggingface.co/OpenTSLM/TeeMoE

Code: https://github.com/OpenTSLM/OpenTSLM-TeeMoE

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: The gist: OpenTSLM TeeMoE introduces a generalist time-series language model that unifies numerical forecasting, context-conditioned prediction, and temporal reasoning by combining independently

Key concepts

OpenTSLM TeeMoE: A generalist time-series language model designed to handle forecasting, contextual prediction, and reasoning simultaneously. It unifies different time-series capabilities into a single model structure.
Low-Rank Experts: Three independently trained capability adapters (experts) that specialize in numerical forecasting, context-conditioned prediction, and temporal analysis. These experts modify the shared backbone using low-rank updates to retain specialized knowledge without retraining the entire model.
LoRA Mixture-of-Experts Controller: A learned controller that decides which expert to use for a given request by assigning weights. It uses a three-way softmax mixture to choose between the numerical decoder, language model head, or one of the three adapted experts based on the input context.
Modular Training Strategy: A staged training approach where each component (aggregation, native forecasting, temporal analysis) is trained separately with its own objectives. This allows for independent revision of specialists while maintaining a unified generalist model.

Summary

The gist: OpenTSLM TeeMoE introduces a generalist time-series language model that unifies numerical forecasting, context-conditioned prediction, and temporal reasoning by combining independently trained low-rank experts within a shared backbone.

Model Architecture and Experts

OpenTSLM TeeMoE is a generalist time-series language model that combines numerical forecasting, context-conditioned prediction, and temporal reasoning over a shared Qwen3.6-27B backbone The model independently trains three low-rank capability adapters for forecast aggregation, native forecasting, and temporal analysis over this shared backbone A learned LoRA mixture-of-experts controller then weights their frozen parameter updates for each request This modular design lets each specialist use its own objectives, optimization settings, and batching strategies, and be revised without retraining the others The controller produces one three-way softmax mixture per request, shared across adapted layers and held fixed for the response

Request Interface and Routing

A request contains an observed series x1:T, a task instruction, and any context or question The model first chooses its expert mixture, then computes the response In the first pass, the backbone runs with all adapters disabled to form the request representation for the controller The controller uses this representation to assign weights to the three adapters and choose the numerical decoder or language-model head A second pass applies the mixed adapters to the full expert inputs

Expert Specialization and Composition

The three independently trained experts are designed for distinct tasks:

Numerical forecasting, where a foundation model aggregation expert refines predictions assembled from external forecasters This involves an XGBoost regressor predicting relative forecast ranks and blending with Toto-FnF
Context-conditioned prediction, where a native forecasting expert learns to generate future values directly from historical observations and textual context
Temporal reasoning, where a temporal analysis expert learns to answer analytical questions about observed signals

To retain specialized capabilities through modular LoRA composition, the backbone is frozen, all three adapters are frozen, and only the controller is trained The composition formula for an adapted matrix in layer l is W'l = Wl + sum e=1 πe∆We,l where ∆We,l is expert e’s low-rank update

Training Strategy

The training follows a staged approach to fit the components sequentially

Aggregation Training involves fitting XGBoost to combine eight selected forecasting models and learning a constant weight that blends its forecast with Toto-FnF The aggregation expert learns a bounded correction from the reference median to the observed future using a weighted SmoothL1 loss and a squared-edit penalty
Native Forecasting Training involves training for one epoch on 20,000 contextual forecasting examples using teacher-forced cross-entropy with forward KL regularization
Temporal Analysis Training involves training for one epoch on 12,000 time-series question-answering examples with response-token cross-entropy and forward and reverse KL regularization
Controller Fitting involves training the controller to minimize Lctrl = sum c(Lc + Hc) where Lc is the prediction loss for capability c and Hc its output-format loss

Performance Evaluation

OpenTSLM TeeMoE achieves strong performance on widely used benchmarks, ranking among the top three on GIFT-Eval by mean MASE rank, Context is Key by RCRPS, and TimeSeriesExam by accuracy The model ranks among the top three on all three benchmarks with a mean MASE rank of 19.990 on GIFT-Eval, an RCRPS of 0.115 on CiK, and 78.552% accuracy on TSE The results show that broad task coverage can coexist with strong predictive performance TeeMoE outperforms the selected individual numerical foundation models on GIFT, achieves lower CiK RCRPS than the locally evaluated time-series language models under the documented transfer protocols, and leads the TSE comparison

Practical Advantages of Modularity

Separate specialization offers practical benefits in training and revising the generalist Joint training must reconcile capabilities’ different objectives and optimization settings in one shared adapter, whereas separate specialization lets each retain its own recipe Separate specialization costs 30.756 H100 GPU-hours, compared to 58.657 for our joint recipe This separation allows an expert and the controller to be retrained while leaving the other experts fixed, so developing one capability need not require training all three again The comparison principle is to retain the specialists’ data exposure and prediction interfaces while replacing separate adapters with shared parameters in joint training

Conclusion

OpenTSLM TeeMoE brings numerical forecasting, context-conditioned prediction, and temporal reasoning into one model, ranking among the top three systems on all three benchmarks Preservation is the clearest benefit as experts trained independently with different objectives retain their strengths when composed within one generalist Learned composition further improves contextual forecasting, with smaller gains on GIFT and TSE The results support a practical route to generalist time-series modeling by training each capability with the objectives and resources it needs, then bringing them together without giving up their strengths or the flexibility to develop them independently

Reproducibility

The codebase is available on GitHub, and model checkpoints are available on Hugging Face Experiments used a cloud-hosted Linux server with eight NVIDIA H100 GPUs with 80 GB of memory each, 128 logical Intel Xeon Platinum 8468 CPUs, approximately 1.5 TiB host RAM, and 3.7 TiB local storage The joint run uses eight H100s for 26,395.480 seconds from the start of its training loop through final checkpointing, giving Cjoint = 8 × 26,395.480/3600 = 58.657 The controller entry measures its training loop, excluding model loading and validation The expert and controller training costs in Table 12 sum to Cmodular ≈ 0.480 + 21.839 + 6.930 + 1.508 ≈ 30.756 The completed joint run uses eight H100s for 26,395.480 seconds from the start of its training loop through final checkpointing, giving Cjoint = 8 × 26,395.480/3600 = 58.657 The controller entry measures its training loop, excluding model loading and validation The XGBoost entry sums the selected full-data and ten cross-fitting fits; it excludes population loading, forecast caching, cross-fitted ensemble prediction, scalar fitting, and preparation of the editor’s training inputs With foundation-model forecasts and request-routing states already cached, full GIFT-Eval evaluation used 8.000 H100 GPU-hours for the composed model; cache construction is additional The composed model’s full CiK evaluation used 5.

Improvements for AI systems

Bold header: Unified Generalist Time-Series Language Model (OpenTSLM TeeMoE)

OpenTSLM TeeMoE can forecast directly from observed time series, reason over textual context and temporal patterns, and synthesize and refine predictions from external numerical forecasting specialists. This allows for a single model to handle numerical forecasting, context-conditioned prediction, and temporal reasoning, unifying capabilities traditionally split across specialized models.

Bold header: Capability-Preserving Mixture-of-Experts Composition

The model employs a learned LoRA mixture-of-experts controller that assigns weights to frozen experts for each request, allowing the system to leverage Independently trained experts retain their strengths when composed within one generalist, demonstrating that broad task coverage can coexist with strong predictive performance.

Bold header: Numerical Forecast Refinement via Aggregation Expert

The aggregation expert refines predictions by constructing a reference forecast through an ensemble of external forecasters and using the shared backbone to refine it, specifically by calculating a correction term: qt,τ = q0,t,τ + gtstdt, which allows for refinement based on candidate disagreement relative to reference uncertainty.

Bold header: Context-Conditioned Forecasting

The native forecasting expert can generate predictions directly from history and text by starting with a continuation that preserves the target history’s level, dynamics, dependence, seasonality, and uncertainty while incorporating context to change it only where supplied context provides supported future evidence.

Bold header: Temporal Reasoning and Analysis

The temporal analysis expert can perform complex reasoning by identifying patterns and anomalies through a mechanism that allows it to answer analytical questions about observed signals, enabling the model to interpret statistical relationships within time-series data.

Abstract

Real-world time-series applications increasingly require models that can handle time series forecasting, context-conditioned prediction, and language-based temporal reasoning. Yet current time-series foundation models remain fragmented across these capabilities: numerical specialists often provide the strongest forecasts, while language-based models offer broader contextual understanding and analysis. A central challenge is to unify these heterogeneous capabilities without reducing their individual performance. We introduce OpenTSLM TeeMoE, a generalist time-series language model that can forecast directly from observed time series, reason over textual context and temporal patterns, and synthesize and refine predictions from external numerical forecasting specialists. We independently train three low-rank experts for forecast aggregation, native forecasting, and temporal analysis over a shared backbone. A learned LoRA mixture-of-experts controller then weights their frozen parameter updates for each request. Our proposed model achieves strong performance on widely used benchmarks for time series forecasting, context-conditioned prediction, and language-based temporal reasoning, ranking among the top three on GIFT-Eval by mean MASE rank, Context is Key by RCRPS, and TimeSeriesExam by accuracy.