GIST: Generator-state-Informed Supervision for Tabular Foundation Models

arXiv:2610.12904 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "GIST: Generator-state-Informed Supervision for Tabular Foundation Models".

Jane: Synthetic generators can support pretraining through conditional label information they retain,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "GIST: Generator-state-Informed Supervision for Tabular Foundation Models". Basically, they tackle a problem where tabular foundation models usually use hard query labels when training, which introduces some extra sampling variability during pretraining.

Jane: It sounds like the core idea here is that intermediate generator states within the model can actually reveal information about label uncertainty even if you only look at a single sampled outcome. They propose something called GIST to use those states to average the query losses over conditional label distributions.

Lu: The paper says GIST jointly designs both task generation and query supervision, using these entropy-calibrated encodings to get targets that are tractable and consistent with the input on those specific branches where labels matter.

Meng: So they're trying to make the supervision targets more robust by leveraging the internal model states instead of just relying on those hard labels we sample once. What they claim is that this averaging process preserves the population objective and doesn't increase gradient covariance at fixed parameters within each generating distribution.

Lalam: I see how they are doing it; they replace the sampled hard-label query loss with its conditional expectation given the model input and the generator information, which fixes the prediction. This allows GIST to average over that remaining label uncertainty whenever those conditional probabilities are tractable.

Tom: That sounds like a clever way to manage that uncertainty you mentioned earlier. So, how do they actually construct these targets?

Jane: Well, in terms of task generation and query supervision, they combine direct conditional targets with entropy-calibrated Gaussian encodings and their posterior targets to build the final target. The construction depends on whether the sampled label goes into input generation or not.

Lu: They create direct conditional targets when the draw doesn't affect the model input or task acceptance, which is nice because it provides supervision right away. But for those label-dependent branches, they represent a sampled class by a category codebook perturbed with Gaussian noise and route all those label-dependent computations through that encoding.

Meng: And then there's this part about entropy budget calibration. They select a fixed scale by bounding the residual entropy with an entropy budget, defining an all-row average upper bound for residual label uncertainty denoted as U(s). They pick the largest finite scale certified by this budget to ensure the encoding retains at least H minus Beta times H of label information on average across all rows before task filtering.

Lalam: So that noise scale controls how much label uncertainty remains after the encoding is observed, and a fixed scale is chosen by bounding that residual entropy with an entropy budget. That selection guarantees that the encoding keeps at least H minus Beta times H of label information on average across all rows before task filtering.

Tom: And then they show statistical guarantees. They claim the conditional supervision preserves the population objective, and they show that the covariance equality follows from the law of total covariance, proving that VH minus VRB is greater than or equal to zero.

Paper summary: Jane: That's a big statement about preserving the population objective while doing this averaging. And for selective supervision, they show VM equals VRB plus E of one minus A times Gamma theta Z, which implies that VH minus VM equals E of A times Gamma theta Z.

Lu: The analysis of gradient covariance decomposition shows that conditional supervision removes Vlabel, while Vtask measures variation across mechanisms compatible with the same visible input. And they show VH is greater than or equal to VRB which is greater than or equal to Cov(GP).

Meng: At the final checkpoints, the prediction head has the largest estimated within-component relative reduction in all six settings, ranging from thirty-seven point seven zero percent to forty-seven point two three percent. The functional decomposition reveals that TFcol accounts for more than half of the net reduction in each setting, accounting for eighty-six point nine five percent of Hard variance in TabICLv2 at L equals one thousand twenty-four.

Tom: So, when we look at the experiments on TabArena and TALENT with matched pretraining schedules, GIST improves mean rank across three architectures without changing the architecture or inference setup at all. And they win sixty-two point one percent of paired dataset-level comparisons against the corresponding hard-label controls <ref:2610.12904#pg1,62.1% of paired dataset-level comparisons against the corresponding hard-label>.

Jane: It means that even when you just use this new supervision strategy, you get better performance on these benchmarks without having to change how the model is built or how it makes a prediction.

Lu: The paper shows that GIST improves mean rank and mean binary ROC AUC in all six combinations of architecture and benchmark, and it wins on a majority of datasets in every combination. TabSwift specifically improves all four reported metrics on both IID benchmarks with paired win rates of sixty-five point eight percent on TabArena and sixty-seven point zero percent on TALENT <ref:2610.12904#pg1>.

Meng: And they also win on fifteen out of eighteen datasets for TabSwift in OOD evaluation across the architecture-specific BeyondArena task sets. That suggests it's pretty versatile for deployment scenarios where you don't have perfect hard labels available during testing.

Tom: So, if I were to put this into my production thoughts, GIST makes that internal generator state information usable by jointly designing task generation and query supervision to get those tractable posterior targets within each fixed generating distribution. It sounds like a solid way to inject structure into the pretraining process.

Jane: Exactly. The paper demonstrates that you can use synthetic information from the generator itself, which is what it produces, to supervise the model in a way that averages out label uncertainty while keeping things stable during training.

Lu: The conclusion is that GIST makes this information usable by jointly designing task generation and query supervision, enabling conditional loss averaging that preserves the population objective within each fixed generating distribution. It motivates considering the supervision a synthetic generator can provide alongside the tasks it produces, with transfer under distribution shift remaining an open direction for future work.

Conclusion: Tom: So we’re wrapping up on GIST, Generator-state-Informed Supervision for Tabular Foundation Models. It boils down to using those hidden states inside the generator to get better supervision for training.

Jane: Right, it’s about taking those internal model states—the things the generator is doing mid-process—and using them to smooth out the noise from sampling hard labels.

Tom: Exactly, and what's interesting is they show that this averaging technique actually keeps the main training objective intact and doesn't make the gradient covariance bigger when you keep your parameters fixed. That’s a clean result for me.

Lu: It means that even if you have some uncertainty about which exact label to use, as long as you average over the conditional distributions correctly, the overall learning path stays stable. The authors prove this through statistical guarantees involving the law of total covariance and showing that VH minus VRB is greater than or equal to zero.

Meng: From a deployment standpoint, that stability is key because we don't want training to become unpredictable at scale. If you can control the variance like that, it makes deploying these foundation models way more reliable without needing constant re-tuning.

Lalam: I’m confident in this because they showed it wins sixty-two point one percent of paired dataset-level comparisons against the hard-label controls on TabArena and TALENT. That win rate isn't just a fluke; it’s consistent across three different model architectures.

Jane: So, what does this mean for someone who doesn't read code? It means we can design these AI models to learn from the *process* of generation, not just the final label, which is much more robust.

Tom: It shifts the focus from just getting a right answer to understanding why that answer is uncertain in the first place. It’s about using synthetic data generated by your own model to guide its own training.

Lu: The authors suggest this approach of joint task generation and supervision might be a way forward, but they also point out that transfer under distribution shift is still an open direction for future work.

Jane: A bit of a caution there, I guess. So the paper shows a strong method for making training more stable and effective right now, but figuring out how to use this information when the data changes a lot is still something they need to explore.

Zhangyang Peng, Jingze Shi, Tongyi Zhang, Yuyu Luo

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Synthetic generators can support pretraining through conditional label information they retain, and GIST makes this information usable by jointly designing task generation and query supervision to

Key concepts

Generator-state-Informed Supervision (GIST): GIST is a framework that links synthetic task creation with conditional query supervision. It uses the internal states of a generator to calculate expected query losses, effectively averaging over label uncertainty while maintaining consistency within each generating distribution.
RaoBlackwellization (RB): RB is a technique used to replace the loss from a sampled hard label with its conditional expectation given the model input and generator information. This allows GIST to condition the prediction on known states, fixing the prediction and enabling averaging over remaining label uncertainty.
Entropy Budget Calibration: This involves setting a limit on how much label uncertainty is allowed in an encoding channel. By bounding residual entropy, GIST selects a fixed scale for Gaussian noise. This ensures that the encoding retains enough label information to maintain performance, balancing noise and fidelity.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes

Summary

Synthetic generators can support pretraining through conditional label information they retain, and GIST makes this information usable by jointly designing task generation and query supervision to obtain tractable posterior targets within each fixed generating distribution.

How it works

GIST is a framework that couples synthetic task generation with conditional query supervision, utilizing RaoBlackwellization (RB) to replace the sampled hard-label query loss with its conditional expectation given the model input and selected generator information. This conditioning fixes the prediction, allowing GIST to average over remaining query-label uncertainty whenever conditional probabilities are tractable. Under a fixed generating distribution, the authors show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution.

Task Generation and Query Supervision

The method involves jointly designing synthetic task generation and query supervision, combining direct conditional targets with entropy-calibrated Gaussian encodings and their posterior targets. The target construction depends on whether the sampled label enters input generation or not, providing direct conditional targets when the draw does not affect the model input or task acceptance. For supported label-dependent branches, GIST represents the sampled class by a category code perturbed with Gaussian noise and routes all label-dependent feature computations through this encoding.

Entropy Budget Calibration

The noise scale controls how much label uncertainty remains after the encoding is observed, and a fixed scale is selected by bounding the residual entropy with an entropy budget. The authors define an all-row average upper bound for residual label uncertainty, denoted as U(s), and select the largest finite scale certified by this budget. This selection ensures that the encoding retains at least H − Bβ(h) ≥ (1 − β)H of label information on average across all rows before task filtering.

Statistical Guarantees and Performance

The conditional supervision preserves the population objective, and the analysis shows that the covariance equality follows from the law of total covariance, proving that VH − VRB ⪰ 0. Furthermore, for selective supervision, it is shown that VM = VRB + E[(1 − A)Γθ(Z)], implying VH − VM = E[AΓθ(Z)]. Experiments across three TFM architectures on TabArena and TALENT show that GIST improves mean rank and wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.

Gradient Variance Reduction

The analysis of gradient covariance decomposition shows that conditional supervision removes Vlabel, while Vtask measures variation across mechanisms compatible with the same visible input, and VH ⪰ VRB ⪰ Cov(GP). At final checkpoints, the prediction head has the largest estimated within-component relative reduction in all six settings, ranging from 37.70% to 47.23%. The functional decomposition reveals that TFcol accounts for more than half of the net reduction in each setting, accounting for 86.95% of Hard variance in TabICLv2 at L = 1024.

Budget Sensitivity

The entropy budget controls the residual label uncertainty permitted in the encoding channel, and experiments on TabSwift show that all three tested budgets improve mean binary AUC and multiclass log loss over Hard on both IID benchmarks. The preferred value varies by benchmark and metric, but gains extend beyond the default budget of β = 0.15 to all three tested values.

Experimental Results

GIST improves mean rank and mean binary ROC-AUC in all six combinations of architecture and benchmark, and it wins on a majority of datasets in every combination. TabSwift improves all four reported metrics on both IID benchmarks, with paired win rates of 65.8% on TabArena and 67.0% on TALENT. GIST wins on 15/18 datasets for TabSwift in OOD evaluation across the architecture-specific BeyondArena task sets.

Conclusion

GIST makes this information usable by jointly designing task generation and query supervision, enabling conditional loss averaging that preserves the population objective within each fixed generating distribution. Improvements across three TFM architectures on TabArena and TALENT support the practical value of this design without changes to model architecture or inference. The results motivate considering the supervision a synthetic generator can provide alongside the tasks it produces, with transfer under distribution shift remaining an open direction.

AI Use Statement

We used generative AI tools to assist with drafting and revising selected parts of the manuscript, including figure and table captions, experimental discussion, and language polishing. Generative AI was also used to provide feedback on the organization of technical material, experimental analysis, and interpretation of reported results. The authors reviewed and revised all AI-assisted content and take full responsibility for the final manuscript.

References

Alan Arazi, Eilam Shapira, and Roi Reichart. Tabstar: A tabular foundation model for tabular data with text fields. In Danielle Belgrave, Cheng Zhang, Laura N. Montoya, Hsuan-Tien Lin, Razvan Pascanu, Piotr Koniusz, Marzyeh Ghassemi, Nancy Chen, Ivan Vladimir Meza Ru´ız, and Arturo Loaiza-Bonilla (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025. Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites. George Casella and Christian P. Robert. Rao-Blackwellisation of sampling schemes. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Xueying Ding and Leman Akoglu. Toward privileged foundation models: LUPI for accelerated and improved learning, 2026. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon-Tabular: Robust and accurate automl for structured data. Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmuller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Muller, and Frank Hutter. Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter-efficient ensembling. Leo Grinsztajn, Klemens Fl ´ oge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jager, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablonskier Shi Bin Hoo. Tabpfn-2.

Improvements for AI systems

Generation of conditional targets using Rao-Blackwellization (RB) preserves population objective while reducing gradient covariance: Proposition 1 shows that this Rao-Blackwellization preserves the population objective and yields no larger gradient covariance under a fixed generating distribution and at fixed model parameters. This allows for more stable training when sampling query labels is variable, as the model learns from conditional expectations instead of hard samples.
Implementation of entropy-budget calibration selects an optimal noise scale: We therefore select the largest scale certified by an entropy budget, adapting it to the task’s class probabilities and codebook, where the budget is defined by: Bβ(h) = β min[H, log 2], binary cap. This ensures that retained information satisfies a specific information guarantee, specifically: the resulting conditional supervision satisfies Equation (31).
Architecture-agnostic performance improvement across TFM variants: GIST improves mean rank and wins on a majority of datasets in all six IID architecture–benchmark combinations, showing gains across TabICLv2, TabSwift, and TabPFN-2.5 without changing the underlying architecture or inference procedure.
Targeted reduction of gradient variance components: Analysis shows that conditional supervision reduces the trace of the clipped batch-gradient covariance; for instance, in TabICLv2 at Stage 2 with L=1024, TFicl has a larger relative reduction than TFcol, 26.85% versus 9.05%, yet contributes less to the whole-model reduction, 2.21 versus 7.87 percentage points. This pinpoints which model components are most affected by the supervision strategy for variance control.
Robustness to training schedule and task length: The method maintains performance across different training stages and table lengths, with results showing that the largest effect at trained checkpoints is consistent with the parameter dependence in Equation (11): before clipping, the logit Jacobian maps conditional label covariance into parameter space. This suggests that GIST's benefits are sustained even as training progresses and context sizes increase.

Abstract

Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining. A key observation is that intermediate generator states can reveal label uncertainty beyond a sampled outcome. We propose Generator-state-Informed Supervision for TFMs (GIST), which uses these states to average query losses over conditional label distributions. GIST jointly designs task generation and query supervision, using entropy-calibrated encodings to obtain tractable, input-consistent targets on supported label-dependent branches. We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution. Under matched pretraining schedules, GIST improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference. Across these benchmarks and architectures, GIST wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.