Probabilistic Circuits for Irregular Multivariate Time Series Forecasting

The gist: CircuITS, a novel architecture for probabilistic IMTS forecasting based on probabilistic circuits, structurally guarantees valid joint distributions and achieves superior joint and marginal density estimation compared to state-of-the-art baselines.

CircuITS Architecture

CircuITS is proposed as a model that leverages the hierarchical nature of Probabilistic Circuits (Choi et al., 2020) to flexibly model the joint distribution over IMTS forecasting queries through recursively stacking sum nodes (mixtures) and product nodes (independencies). This structure allows it to capture intricate correlations and independent factors alike, while strictly guaranteeing marginalization consistency. The recursive Sum-Product Network (SPN) iteratively builds the joint density by alternating between product layers (independence assumptions) and sum layers (mixture projections), which involves a recursive step where the new joint state is computed as: log ϕ(c)k = LSE log Ψ + logW(c)·,k. The final joint likelihood is given by a weighted sum of the final K components using the root weights wroot: p(y Q, X) = X K k=1 wroot k · ϕ(C)k.

Marginalization Consistency Guarantee

A critical contribution of CircuITS is that it structurally guarantees marginalization consistency, ensuring valid and non-contradictory predictions for any subset of variables. This is proven by showing that the circuit structure must be fixed and aggregate meaningful subsets of variables, ensuring both the graph topology and the mixing weights retain consistent semantic interpretations regardless of the query. The proof demonstrates that marginalization consistency holds up to the leaves if Property 1 (leaf nodes represent a single input channel) is met, Property 2 (marginal leaf distributions only depend on the query subset), and Property 3 (weights of sum nodes do not depend on the query set Q) are satisfied. Furthermore, Proposition A.2 confirms that CircuITS’s leaves are marginalization consistent, relying on the Gaussian Copula combined with univariate Deep Sigmoidal Flows (DSF) to model the leaf distributions.

Encoder and Context Generation

The encoder architecture is designed to infer both the PC weights and the embeddings of the forecasting queries. Observation representation involves projecting discrete observations into a latent feature space of dimension D, where each observation embedding hn is computed as hn = ϕ(tn)∥ Echan chan∥ yn. Intra-Channel Aggregation uses Multi-Head Attention (MHA) with static learnable embeddings Eproto to group observations by channel, producing a matrix Hchan. Inter-Channel Interaction captures dependencies between variables using a Self-Attention layer, resulting in the updated channel representations H˜ = Hchan + MHA(Hchan, Hchan, Hchan). The circuit weights are parameterized via cross-attention on H˜ using learnable representations Ecw, and the root weights wroot and W(c) only depend on X and not on Q, which is necessary to ensure CircuITS is marginalization consistent.

Leaf Distribution Modeling

The leaf distributions model the univariate marginals of channels using a Gaussian Copula combined with univariate Deep Sigmoidal Flows (DSF). The DSF approximates the cumulative distribution function (CDF) via a composition of L monotonic layers, where the l-th layer operation is defined as fl(x) = σ(Alx + bl). To condition the flow on the input context, a two-layer MLP maps the query embedding e k to instance-specific flow parameters θ k. The correlation matrix Rc,k for each channel c and mixture component k is computed using scaled inner products of features v c,k = tanh(MLPcorr(e k)), ensuring Rc,k is positive definite. The log-likelihood of the leaf distribution is then defined as log pc,k = X Qc i=1 log p(yie ki) z univariat marginals + L GC(F(yi, θki), Rc,k) copula density.

Computational Efficiency and Performance

CircuITS is designed to be computationally efficient by decoupling inter-channel dependencies and performing matrix operations only at the leaf level for each channel independently. The recursive Sum-Product Network (SPN) aggregation can be reformulated as a sequence of matrix multiplications in the log-semiring, allowing parallel prefix scan algorithms to reduce the time complexity from linear O(C) to logarithmic O(log C) on parallel hardware. Despite this efficiency, the cubic scaling with respect to the number of mixture components K remains a bottleneck, leading researchers to keep K ≤ 4 in experiments. In real-world scenarios where query distribution is uneven, CircuITS can be slower than MOSES and ProFITi as it groups forecasting targets by channels.

Experimental Validation

Experiments validate CircuITS on a synthetic bifurcation task where it successfully isolates the upward and downward trends with only 2 components per channel, whereas MOSES fails even with 16 components. On real-world datasets (USHCN, PhysioNet-2012, MIMIC-III, MIMIC-IV), CircuITS achieves superior joint density estimation compared to its baselines ProFITi and MOSES across majority of scenarios. The model's performance is quantified using the normalized joint Negative Log Likelihood (njNLL) and marginal Negative Log Likelihood (mNLL), where CircuITS consistently achieves the lowest njNLL in the vast majority of scenarios. Furthermore, evaluation using Energy Score and CRPS shows that CircuITS maintains competitive quality across all tested datasets. The sensitivity analysis indicates moderate robustness to variations in K, suggesting K = 2 or K = 3 is sufficient for the evaluated task.

Impact Statement

This work introduces a probabilistic forecasting architecture that structurally guarantees marginalization consistency for irregular multivariate time series. This property significantly enhances reliability in safety-critical domains, such as healthcare and climate science, by preventing contradictory forecasts that can lead to erroneous decision-making. The model's structural guarantee ensures that the predicted marginal distribution of a query subset must match the integrated joint distribution, which is essential for reliable stochastic process approximation.

References

Anderson, J., Belkin, M., Goyal, N., Rademacher, L., and Voss, J. The More, the Merrier: The Blessing of Dimensionality for Learning Large Gaussian Mixtures. In Proceedings of The 27th Conference on Learning Theory, pp. 1135–1164. PMLR, May 2014.

Ashok, A., Marcotte, E., Zantedeschi, V., Chapados, N., and ´ Drouin, A. TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series. In The Twelfth International Conference on Learning Representations, October 2023.

Bilos, M., Sommer, J., Rangapuram, S. S., Januschowski, T., ˇ and Gunnemann, S. Neural Flows: Efficient Alternative ¨ to Neural ODEs. In Advances in Neural Information Processing Systems, volume 34, pp. 21325–21337. Curran Associates, Inc., 2021.

Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(1):6085, April 2018 ISSN 2045-2322.

Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

Choi, Y., Vergari, A., and Van den Broeck, G. Probabilistic circuits: A unifying framework for tractable probabilistic models. UCLA. URL: http://starai.cs.ucla.edu/papers/ProbCirc20.pdf, pp. 6.

De Brouwer, E., Simm, J., Arany, A., and Moreau, Y. GRU-ODE-Bayes: Continuous Modeling of SporadicallyObserved Time Series. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.

Drouin, A., Marcotte, E., and Chapados, N. TACTiS: ´ Transformer-Attentional Copulas for Time Series. In Proceedings of the 39th International Conference on Machine Learning, pp. 5447–5493. PMLR, June 2022.

Durichen, R., Pimentel, M. A. F., Clifton, L., Schweikard, ¨ A., and Clifton, D. A Multitask Gaussian Processes for Multivariate Physiological Time-Series Analysis.

1. No trade-off between expressivity and consistency: CircuITS structurally guarantees marginalization consistency, allowing it to model complex joint dependencies where they exist, while efficiently factoring into independent components where they do not. This enables reliable forecasting by ensuring that predictions are consistent regardless of which subset of variables is queried.

2. Superior joint density estimation: The paper demonstrates that CircuITS achieves superior joint and marginal density estimation compared to state-of-the-art baselines, including ProFITi and MOSES, leading to more accurate quantification of uncertainty in forecasts.

3. Handling complex dependencies: By using a recursive Sum-Product Network (SPN), CircuITS can flexibly model the joint distribution over IMTS forecasting queries by recursively stacking sum nodes (mixtures) and product nodes (independencies), capturing intricate correlations and independent factors alike.

4. Robustness to query configuration: The architecture avoids the failure of other models where weights depend on the query, as CircuITS ensures that the weights [of the sum nodes] do not depend on the query set Q, which is necessary for consistency.

5. Efficient computation for high dimensions: By reformulating the SPN aggregation as a sequence of matrix multiplications and utilizing parallel prefix scan algorithms, CircuITS can reduce time complexity from linear to logarithmic, achieving O(log C) sequential steps with sufficient parallel threads on GPU hardware.

6. Improved uncertainty quantification: The model provides better estimates for both joint density (njNLL) and marginal density (mNLL), which is crucial for decision-making in high-stakes domains like healthcare, as evidenced by its competitive performance across medical datasets.

7. Adaptive leaf modeling: The leaf distributions use Conditional Deep Sigmoidal Flows (DSF) combined with a Gaussian Copula, allowing the model to learn complex, non-Gaussian marginals while maintaining the structural guarantees of consistency through parameter stability.

Joint probabilistic modeling is essential for forecasting irregular multivariate time series (IMTS) to accurately quantify uncertainty. Existing approaches often struggle to balance model expressivity with consistent marginalization, frequently leading to unreliable or contradictory forecasts. To address this, we propose CircuITS, a novel architecture for probabilistic IMTS forecasting based on sum-product networks (SPNs). Our model is flexible in capturing intricate dependencies between time series channels while structurally guaranteeing valid joint distributions. Experiments on five real-world datasets demonstrate that CircuITS achieves superior joint and marginal density estimation compared to state-of-the-art baselines.