A Mixture-of-Experts Framework for Multi-Stream Quickest Change Detection

The gist The MoE-based QCD algorithm for multi-stream data outperforms the RWS scheme when considering different mixture coefficients and weighting functions.

How it works

1. A Mixture-of-Experts (MoE) framework is proposed where multiple experts, each producing a monitoring statistic, combine their contributions via a weighted sum to form a global QCD statistic: Gen = Sum e π(e)n G(e)n<ref:2610.20876#pg6> This framework is sufficiently general to accommodate different choices of experts and the corresponding weights<ref:2610.20876#pg4>.

2. In the multi-stream QCD problem, each stream's Shiryaev-Roberts statistic is treated as an expert, and the mixture is defined as G(w)n = Sum k w(k)R(k)n<ref:2610.20876#pg6> This generalizes the RWS scheme to allow for non-linear weights<ref:2610.20876#pg4>.

3. The stopping rule is defined as T(w) = inf[n ≥ 1: G(w)n ≥ Aw]<ref:2610.20876#pg6> where Aw is selected to satisfy a prescribed average run length to false alarm constraint, E∞ h T(w)i ≥ γ<ref:2610.20876#pg4>.

4. The paper investigates different mixture coefficients or rank-weighting functions such as log(1 + k), √k, k, k squared, k cubed, and k 4<ref:2610.20876#pg6> These choices lead to random data-dependent mixture coefficients because the ranks of the Shiryaev-Roberts statistics are time-varying<ref:2610.20876#pg6>.

Comparison with Conventional Methods

The MoE approach is compared against the rank-weighted Shiryaev–Roberts scheme studied in [20]<ref:2610.20876#pg6> The results show that the best choice of mixture coefficients depends on the number of affected streams<ref:2610.20876#pg6>. When m = 3, the logarithmic weight log(1 + k) gives the smallest observed delay<ref:2610.20876#pg6> For m = 5, 8, 10, and 20, the linear weight w(k) = k gives the smallest delay<ref:2610.20876#pg6>. When the number of affected streams increases, higher-order polynomial weights become more effective<ref:2610.20876#pg6>.

Performance Across Sparsity Regimes

The best-performing mixture function depends strongly on m<ref:2610.20876#pg6>. When the number of affected streams is small, the linear weighting function performs well<ref:2610.20876#pg6> For example, at m = 5, 8, 10, and 20, the smallest detection delay among the considered polynomial weights is attained by w(k) = k<ref:2610.20876#pg6>. In contrast, when more streams are affected, higher-order polynomial weights become increasingly effective<ref:2610.20876#pg6>. The quadratic weight w(k) = k squared gives the smallest delay for m = 30,..., 70<ref:2610.20876#pg6>, while k cubed and k 4 achieve the smallest delays at m = 80 and m = 100, respectively<ref:2610.20876#pg6>.

Comparison with Score-Based Statistics

The MoE framework is also applied to score-based statistics, showing that the same qualitative behavior is observed beyond likelihood-ratio-based statistics<ref:2610.20876#pg6> In the GB-RBM data generation experiment, a local score-based statistic is updated recursively as Rk,n = (1 + Rk,n−1) exp (λY (Xk,n))<ref:2610.20876#pg6>. The comparison of rank-weighting functions for score-based statistics shows that the linear weight w(k) = k consistently provides the best or nearly the best performance when only a small number of streams are affected<ref:2610.20876#pg6>. As the number of affected streams increases, however, higher-order polynomial weights become increasingly effective and yield substantially shorter detection delays<ref:2610.20876#pg6>.

Conclusion

The results indicate that no single fixed rank-weighting function is uniformly optimal across different post-change regimes<ref:2610.20876#pg6> While the conventional linear weight performs well when relatively few streams are affected, higher-order polynomial weights achieve noticeably shorter detection delays when a larger number of streams is affected<ref:2610.20876#pg6>. This suggests that the choice of rank-weighting function is an important design component of rank-based fusion and should depend on the underlying change scenario<ref:2610.20876#pg6>. The paper shows the promise of an MoE framework for multi-stream QCD and for QCD in general<ref:2610.20876#pg6>. An important direction for future research is to develop adaptive procedures that automatically select or combine different weighting functions online according to the observed data<ref:2610.20876#pg6>.

--- Page 1 ---

A Mixture-of-Experts Framework for Multi-Stream Quickest Change Detection Wuxia Chen, Arman Adibi, Taposh Banerjee, and Vahid Tarokh Abstract A Mixture-of-Experts (MoE) framework is proposed for quickest change detection (QCD) where there are multiple experts, and each expert produces a sequence of statistics for change detection. A change is declared when a weighted sum of these statistics crosses a threshold<ref:2610.20876#pg6> This framework is then applied to the problem of multi-stream QCD to obtain a family of powerful algorithms that outperform the best known algorithms in the literature<ref:2610.20876#pg6>. In the multi-stream QCD problem, a decision-maker observes multiple streams of observations<ref:2610.20876#pg6>. At the change point, the distribution of an unknown subset of the streams changes<ref:2610.20876#pg6> The optimal algorithm based on a generalized likelihood ratio is computationally expensive<ref:2610.20876#pg6>. One of the most computationally and performance-efficient algorithms in the literature is the rank-weighted sum (RWS) scheme based on Shiryaev-Roberts statistics<ref:2610.20876#pg6> It is shown in the paper that the MoE-based QCD algorithm for multi-stream data outperforms the RWS scheme<ref:2610.20876#pg6> This is demonstrated through simulation results on real-valued Gaussian mean shift data and high-dimensional data from Gaussian–Bernoulli Restricted Boltzmann Machines (GB-RBMs)<ref:2610.20876#pg6>. Index Terms Multi-Expert Fusion, Multi-Stream Quickest Change Detection, Rank-Weighted Fusion, Score-Based Change Detection<ref:2610.20876#pg6>

--- Page 3 ---

In this work, we first propose a MoE architecture for the general QCD problem, not necessarily for multi-stream QCD<ref:2610.20876#pg6> We believe that this architecture can be used to find powerful algorithms applicable to a wide range of applications<ref:2610.20876#pg6>. The proposed QCD-MoE architecture naturally raises the question of how the experts and their weights should be selected, with the possibility of the weights being adapted online based on the past observed data<ref:2610.20876#pg6>. However, the focus of this work is merely to establish the effectiveness of the MoE framework<ref:2610.20876#pg6>.

--- Page 4 ---

A detection procedure is a stopping time T adapted to the filtration generated by the observations<ref:2610.20876#pg6> In the formulation of Lorden [7], the false-alarm performance of T is measured by the average run length (ARL) to false alarm, ARL(T):= E∞[T]<ref:2610.20876#pg6> The delay of the stopping time is measured by the Worst-case Average Detection Delay (WADD) metric: WADD(T) = sup ν ess sup Eν[(T − ν + 1)+Fν−1]<ref:2610.20876#pg6>

--- Page 5 ---

The MoE-QCD stopping rule is T = inf[n ≥ 1: Gen ≥ A], where A is a prescribed detection threshold to control the rate of false alarms<ref:2610.

1. A Mixture-of-Experts (MoE) framework can be applied to multi-stream Quickest Change Detection (QCD) by treating each stream statistic as an expert, allowing for a fusion rule or mixture is used to obtain a global statistic. This enables the system to adapt its decision strategy based on the change scenario, as demonstrated by showing that no single fixed rank-weighting function is uniformly optimal across different post-change regimes.

2. The improved AI system can achieve superior performance in multi-stream monitoring by dynamically selecting or combining experts using different mixture coefficients. Specifically, the system can switch between a logarithmic, linear, and polynomial weights based on the observed sparsity of the change, as shown by comparing results where the best choice of mixture coefficients (among the categories chosen) depend with the number of affected streams.

3. The system will be capable of handling non-Gaussian high-dimensional data, such as that from Gaussian–Bernoulli Restricted Boltzmann Machines (GB-RBMs), effectively using score-based statistics. This is achieved by employing a Hyvärinen score and updating local statistics recursively: "Rk,n = (1 + Rk,n−1) exp (λY (Xk,n)), where λ > 0 is the scaling parameter proposed in [27]."

4. The improved system can be optimized for real-time decision-making by utilizing time-varying mixture coefficients derived from historical data. The paper suggests that the optimal mixture coefficients will be a time-varying non-linear function of all the past history of the data collected, which, if approximated by an artificial neural network, allows for adaptive fusion rules.

A Mixture-of-Experts (MoE) framework is proposed for quickest change detection (QCD) where there are multiple experts, and each expert produces a sequence of statistics for change detection. A change is declared when a weighted sum of these statistics crosses a threshold. This framework is then applied to the problem of multi-stream QCD to obtain a family of powerful algorithms that outperform the best known algorithms in the literature. In the multi-stream QCD problem, a decision-maker observes multiple streams of observations. At the change point, the distribution of an unknown subset of the streams changes. The optimal algorithm based on a generalized likelihood ratio is computationally expensive. One of the most computationally and performance-efficient algorithms in the literature is the rank-weighted sum (RWS) scheme based on Shiryaev-Roberts statistics. It is shown in the paper that the MoE-based QCD algorithm for multi-stream data outperforms the RWS scheme. This is demonstrated through simulation results on real-valued Gaussian mean shift data and high-dimensional data from Gaussian--Bernoulli Restricted Boltzmann Machines (GB-RBMs).