Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Gate-Based Feature Importance in Simulation-Based Inference for EEG".

Marcus: Here's the gist of the paper. The gist The gate was internally consistent and reproducible across activation functions; it simply tracked something other than what ablation tracked.

Ines: Simulation-Based Inference Context Simulation-based inference (SBI) provides a powerful way to fit mechanistic models to data, but offers little insight into which information in the data shapes the inferred parameter values Simulation-based inference (SBI) completely sidesteps evaluating intractable likelihoods by training a neural density estimator directly on simulation outputs, learning an amortized mapping from data features to a posterior over parameters In scientific applications of SBI, it becomes crucial to understand how each feature of the data contributes to parameter inference Normally, a feature’s contribution is established only after training; the gold standard in SBI is retrain-based ablation, in which the model is retrained with one feature excluded and the change in the inferred posterior is quantified, a causal measure by construction An appealing, cheaper alternative is to learn the weighting itself by incorporating a small differentiable gate, trained end-to-end with the rest of the network Indeed, gates of exactly this form are reported in various settings: as electrode importance in EEG deep learning

12, 13: ; and, in tabular learning and forecasting, the reading is built into the methods themselves, through learned input masks

14: , attention-based feature selection

15: , and variable-selection layers

16: In SBI, however, the idea appears not to have been taken up (although it has been suggested

17: ), and we believe it could be attractive in principle, since the standard ablation test costs one full retraining per feature The researchers compared the gating and ablation measures of importance directly in a Jansen–Rit EEG model (Figure 1) Neural posterior estimators (NPEs) were trained over four mechanistic parameters: connectivity strength, input drive, the joint excitatory/inhibitory timescale-scaling factor, and inhibitory synaptic gain Two gated variants, independent (sigmoid) and competing (softmax) per-feature weighting, were compared against an ungated baseline in terms of model performance The gold-standard measure of feature importance was obtained by retraining the full pipeline from scratch with each feature excluded in turn and measuring the resulting drop in posterior accuracy This procedure, rather than a post-hoc masking heuristic, as it measures each feature’s causal contribution by construction All experiments were run on a single consumer NVIDIA RTX 4050 Laptop GPU Key Findings on Feature Importance Disagreement Across nine independent training seeds, ablation identified total signal power as the most important of the seven features by a wide margin: excluding it dropped mean R2 by 0.210 (SD 0.009), roughly five times the drop from the next-largest feature Testing each feature’s own ablation effect against the null with the exact sign-flip test described in Methods, six of the seven features had an effect significantly different from zero (Table 1) In contrast, the gate did not favor total signal power In fact, it assigned the feature the lowest or second-lowest weight under both activations (Table 1), while its two most-favored features, dominant frequency and Hjorth mobility, had ablation effects an order of magnitude smaller This disagreement was systematic We computed the Spearman rank correlation between ablationbased and gate-based importance separately for each seed (seven features per seed) and tested the seed-averaged correlation against zero with the exact sign-flip test Independent (sigmoid) gating gave ρ = −0.607 and competing (softmax) gating gave ρ = −0.556, both at p = 0.0039 Model Performance and Gate Reliability Gating cost little in raw accuracy: mean R2 was 0.791 without gating, 0.792 with per-feature (sigmoid) gating, and 0.761 with competing (softmax) gating Credible-interval coverage was close to the nominal 50%, 90%, and 95% levels under all three configurations, thus showing no evidence of systematic over- or under-confidence from either gating variant Moreover, ordering the four parameters by per-parameter R2 resulted in an identical ranking under every configuration (Appendix D) Our main finding was that ablation-based and gate-based feature importance were negatively correlated for both gating variants This disagreement is arguably more pronounced than a weak correlation, such as, for example, that reported by Jain and Wallace

32: in a natural language processing setting Our results therefore reinforce the broader conclusion that mechanisms purely optimized for predictive accuracy should not automatically be interpreted as reliable indicators of feature importance A gating mechanism optimized purely for predictive accuracy is not thereby guaranteed to possess explanatory power Practitioners using learned gating or attention as a cheap stand-in for feature-importance analysis in scientific, simulation-based inference pipelines should treat this as a caution rather than a shortcut We therefore recommend that learned weighting be validated against a causal measure, such as the retrain-based ablation used here, before it is reported as importance Responsible Use Statement The risk this work identifies is misplaced trust: a coherent-looking gate can support an incorrect scientific conclusion under the appearance, rather than the substance, of mechanistic justification We therefore recommend that learned weighting be validated against a causal measure, such as the retrain-based ablation used here, before it is reported as importance Acknowledgments and Disclosure of Funding The authors declare no competing interests References

1: Ben H.

Marcus: Jansen and Vincent G.

Ines: Rit.

Title and authors: Ines: So we're looking at this paper today called "Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale." It’s about how we try to figure out which parts of the input data are actually important when we use these simulation-based inference methods.

Marcus: Exactly. Basically, the authors are looking at whether a simple learned gate mechanism can tell us which features in the data matter, and they're calling it a cautionary tale because it might be misleading us.

Yuki: From my side, I’m curious if this applies to how we look at population data. When we use these methods to infer parameters from simulated EEG data, does this mean we can trust what the AI is telling us about which underlying biological processes are driving the signal?

Ines: That’s a good starting point, Yuki. The paper sets up a Jansen–Rit model, which is like a mathematical representation of interacting neuronal populations. They train neural posterior estimators over four parameters: connectivity strength, input drive, the joint excitatory/inhibitory timescale-scaling factor, and inhibitory synaptic gain.

Marcus: And they use seven summary statistics per channel to feed into this system. The core idea here is using a small MLP to produce a weight for each of those seven features—a sigmoid or softmax gate—to scale them before they go into the density estimator.

Yuki: But the real question for me, as someone who studies populations, is whether these learned weights actually align with what we expect from biological constraints when we look at larger samples.

Ines: Well, the authors are comparing this learned gate approach against a gold standard measure of importance. This gold standard is retraining the whole pipeline from scratch every time and excluding one feature at a time to see how much the posterior accuracy drops.

Marcus: That’s where they get their main result. They found that ablation—that process of retraining with each feature excluded—was much more consistent across different training runs than their learned gating mechanism was.

Yuki: So, if the ablation measure shows one thing but the gate suggests something else, what does that imply for our understanding of these complex dynamical systems?

Ines: It implies that mechanisms purely optimized to get predictive accuracy might not be reliable indicators of which features are truly important. The paper says a gating mechanism optimized purely for predictive accuracy isn't guaranteed to possess explanatory power.

Marcus: That’s the core caution they want listeners to hear: don't just trust the gate because it looks like it’s working, even if the rankings look moderately similar between the sigmoid and softmax variants.

Yuki: I wonder how this affects how we interpret our large-N studies. If we rely on these cheap interpretability tools, are we risking misidentifying fundamental biological drivers in a population context?

Ines: The paper suggests that when you use learned weighting as a cheap stand-in for feature importance analysis, practitioners should treat it as a caution rather than a shortcut. They recommend validating learned weighting against that causal measure, the retrain-based ablation they used here, before reporting it as importance.

Title and authors: Marcus: It’s about moving away from post-hoc heuristics to something more grounded in causality when interpreting these simulation-based results. The authors found that for instance, total signal power was the most important feature according to ablation, but the gate didn't favor that same feature at all.

Yuki: That’s a big difference. If the gate assigns low weight to features that ablation says are crucial, then we have no real way of knowing what’s actually driving the parameter estimation in this EEG model.

Ines: The disagreement is systematic across nine independent training seeds, and they computed Spearman rank correlation between the two measures and tested it against zero using a sign-flip test, finding correlations of negative values like rho = −zero point six zero seven for the sigmoid gate.

Marcus: And even though the raw model performance didn't change much—mean R2 was zero point seven nine one without gating, zero point seven nine two with the sigmoid variant—the discrepancy in feature importance ranking is what’s concerning them most statistically.

Yuki: So for a listener who only cares about population genetics, this means we need to be very careful when we apply these techniques to our own data, because the cheap weighting might just be pointing us in the wrong direction regarding the underlying biology.

Ines: Exactly. The paper emphasizes that this disagreement is arguably more pronounced than what you might see in other fields, like natural language processing settings reported by Jain and Wallace. They’re advocating for a more rigorous validation process before we start using these learned weights as a substitute for causal testing.

Marcus: So, to wrap up on the methodology, the paper compared independent sigmoid gating and competing softmax gating against an ungated baseline in terms of model performance, but their key finding is that ablation-based and gate-based importance were negatively correlated for both variants.

Yuki: That negative correlation suggests that when you look at what’s important by one method, it’s often the opposite of what the other method highlights. It really underscores how fragile these learned approximations can be in scientific inference.

Ines: The overall message of "Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale" is that predictive performance does not automatically mean interpretability or feature importance is reliable. We need causal measures, like retrain-based ablation, to validate any learned weighting before we claim it tells us anything meaningful about the underlying mechanism.

Marcus: It’s a strong nudge for anyone using these cheap ways to get insights into simulation-based models—treat the gate as an educated guess, not a proven fact about what drives the system.

Yuki: I think this is important because it reminds us that when we apply these tools to real biological systems, those learned weights could be reinforcing incorrect interpretations of how those populations interact over time.

Ines: So, remember, when you see a neat feature importance plot from a gate in an SBI pipeline, ask yourself: have I checked this against a causal measure like ablation? That’s the caution the paper is pushing for.

The paper's summary: Ines: So, we just looked at how the authors used this gate mechanism to measure feature importance in their simulation models.

Marcus: Right, and what they found was that the learned weighting didn't match up with what a more rigorous test showed.

Ines: Exactly. They compared their gating method against retraining the whole pipeline with each feature excluded, and that causal measure showed a disagreement.

Marcus: The paper says the gate assigned total signal power the lowest or second-lowest weight, while ablation found it was actually most important by a wide margin.

Ines: And they quantified this disagreement using Spearman rank correlation, finding negative numbers for both the sigmoid and softmax gating variants.

Marcus: It sounds like there's a systematic difference between what the AI learned to be important and what we can measure by removing features one by one.

Ines: That’s the cautionary part, Marcus. The authors are making a strong point that mechanisms optimized purely for predictive accuracy aren't automatically reliable indicators of feature importance.

Marcus: They found that even though raw model performance didn't really change much with the gate on or off, the ranking of which features mattered was flipped in a systematic way.

Ines: So, they’re warning people using learned gating to treat it as a caution rather than just a shortcut for feature importance analysis.

Marcus: They say that if you’re using learned weighting in simulation-based inference pipelines, you need to validate it against something causal like the retrain-based ablation before you report anything.

Ines: It’s about making sure that what looks coherent on the surface doesn't actually reflect the true underlying biology of the system.

Marcus: They stress that this isn't just a minor correlation; they found a pretty strong negative correlation between those two importance measures for both gating setups.

Ines: So, as computational biologists, we need to be careful not to mistake model optimization for understanding causal drivers in complex systems.

Marcus: And I think Yuki’s point about population genetics fits here because if our small-scale models are already showing this kind of confusion over feature importance, applying it to larger cohorts could lead us astray.

Ines: Exactly. It reminds us that when we apply these cheap interpretability tools, we need a much stronger grounding in causal validation for any scientific claim about what’s driving the system.

Marcus: So if you’re working with simulation-based inference, don't just trust the gate; run the ablation test to see what’s actually causing that performance drop.

Ines: And that sets up a good discussion for next time because we need to look at how these kinds of learned weights affect actual parameter estimation.

The paper's improvements: Marcus: So, we just talked about how the authors found a systematic disagreement between their learned gate importance and the causal ablation measure for feature importance in this EEG model.

Ines: Right, and they're not just stopping there with that finding; they actually suggest some improvements to how we do this kind of feature selection.

Marcus: They propose replacing the simple learned gating mechanism with a method that is explicitly designed to recover ground truth variables when applied to synthetic data.

Ines: That makes sense because the whole point was that a gate optimized for accuracy might just be pointing us in the wrong direction regarding what’s actually driving the biology.

Marcus: They want to move away from those simple learned weights and validate them against something causal before we report anything as important.

Ines: It sounds like they are advocating for a system where feature selection methods are built into the pipeline to ensure they actually pick variables that matter biologically, not just mathematically.

Marcus: They draw inspiration from findings that show architectures built for feature selection can sometimes recover known relevant variables on synthetic data.

Ines: So, it’s about making the interpretability layer more robust so it doesn't just mimic accuracy without actually capturing the causal structure of the EEG model.

Marcus: The implication is that if we want to use learned weights as a cheap stand-in for feature importance in simulation-based inference, we should treat that weight as a caution rather than a shortcut.

Ines: They’re pushing for better validation methods because the current approach, while cheap and fast, doesn't guarantee explanatory power about the underlying mechanisms.

Marcus: This speaks to the larger issue of interpretability in these models—we need tools that respect biological constraints like nonnegativity and energy efficiency.

Ines: It means that when we build these systems, we need to ensure our interpretability methods are actually tied to those physical or biological rules, not just statistical noise.

Marcus: So the paper’s suggestion is this: don't just trust the gate because it looks coherent; validate it against a causal measure like ablation first.

Ines: And that leads us into what this means for population studies because if we can’t trust feature importance in a single model, how do we apply those ideas to whole cohorts?

Conclusion: Ines: So we're wrapping up on "Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale."

Marcus: Basically, the main message is that a gate optimized for accuracy doesn't automatically mean it’s showing us what’s actually important about the system.

Ines: Exactly. The paper shows that ablation—the causal measure—is more consistent across different training runs than the learned gating mechanism was.

Marcus: That disagreement, quantified by negative correlation, means we can't just trust the ranking produced by a learned weight when interpreting simulation results.

Ines: They are urging practitioners using these cheap weighting methods to treat it as a caution rather than just a shortcut for feature importance analysis.

Marcus: So if you’re using learned gating in simulation-based inference, you need to run that retrain-based ablation test before you report any conclusions about feature importance.

Ines: It sounds like they’re pushing for a more rigorous validation process because the current approach doesn't guarantee explanatory power about the underlying biological drivers.

Marcus: That ties into our bigger picture with population genetics; if we can’t trust feature importance in a single model, how do we apply those ideas to whole cohorts?

Ines: Right. And that brings up a question for next time about how these kinds of learned weights affect actual parameter estimation and whether they align with biological constraints.

Marcus: We need to keep asking that question when we look at any new interpretability tool, because the risk is misinterpreting the substance under the appearance of justification.

Ines: I agree. It’s about moving away from post-hoc heuristics toward methods that are grounded in causality when we’re looking at these complex dynamical systems.

Marcus: So for anyone working on simulation-based inference, remember that predictive performance doesn't automatically equal reliable feature importance.

Yuki: From a population genetics standpoint, this means we need to be even more cautious about using simplified interpretability tools to infer the true history of a species across large samples.

Ines: It reinforces the idea that biological constraints should guide our interpretation, not just statistical optimization.

Marcus: So that’s it for this episode on "Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale." We'll be back next week to look at how these same ideas apply to disentangling functional cell types.