GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

The gist GeneICL introduces a 4.2M-parameter tabular foundation model that combines transcriptomics-aware pretraining with in-context learning to predict clinical outcomes from bulk gene expression data, outperforming existing models by leveraging a semi-synthetic prior and an efficient recurrent architecture

GeneICL Architecture and Pretraining

GeneICL operates as a parameter-efficient tabular foundation model pretrained on a semi-synthetic prior built from measured bulk expression profiles, where its expression profiles are sampled from a variational autoencoder trained on real bulk RNA-seq data, while its labels are generated by random synthetic functions of these profiles The model uses a recurrent, weighttied in-context learner to predict query samples from labeled support examples The input features are standardized and projected onto their principal components (PCs), with both transformations fitted on the support set alone, which reduces the number of feature tokens from G to at most ns-1 The encoder alternates between column- and row-attention across three transformer blocks, using a bottleneck over [CLS] tokens for row attention The model uses a single pretrained model that supports both regression and classification through separate output heads, keeping the parameters fixed for every downstream task

Pretraining Prior Construction

The pretraining prior is constructed in two stages to capture transcriptomic structure while allowing for unlimited supervised tasks during pretraining First, a variational autoencoder (VAE) is trained on human bulk RNA-seq profiles from ARCHS4, which comprises 547,829 human RNA-seq profiles after preprocessing Synthetic expression profiles are generated as xi = Dϕ(zi), i = 1,..., n, allowing for sampling an effectively unlimited number of expression datasets while retaining structure learned from real transcriptomic data Second, for each generated expression dataset, a supervised prediction problem is sampled by generating latent task scores sik = gψk(ui,J), k = 1,..., K Classification and regression targets are constructed from the same latent score functions to jointly supervise GeneICL’s classification and regression capabilities The training objective minimizes L = CEPˆ Q, y clf Q + LrankPˆ Q, y clf Q + MSEyˆreg Q, y reg Q

Inference-Time Learning and Adaptation

GeneICL enables right-censored survival prediction via a training-free reduction to regression using Cox partial-likelihood residuals For survival outcomes with observed time ti and event indicator δi, the model regresses on null-model martingale residuals ri = δi − Hˆ0(ti), Hˆ0(t) = X τk≤t dk/nk The residual ri is the gradient of the Cox partial log-likelihood with respect to the log-risk fi at f = 0, so predicting it is a functional gradient step from the null model GeneICL uses this as an ordinary regression target, fQ = fθ(XS, rS, XQ,reg), and predicts survival zero-shot without structural modifications or dedicated retraining

Performance and Evaluation

GeneICL achieves the best overall rank among evaluated foundation models and tuned baselines with up to 387× fewer parameters than TabFM In classification and regression tasks, GeneICL matches or outperforms general-purpose tabular foundation models with up to 387× more parameters The model's performance is evaluated on a benchmark of 80 clinical outcome-prediction tasks spanning classification, regression, and survival GeneICL achieves the second-best overall average rank across classification and regression, surpassed only by its inference-time ensemble For survival prediction, GeneICL-Cox achieves the highest aggregate C-index and best D-calibration while remaining within 0.001 of the best integrated Brier score without any survival-specific pretraining

Computational Efficiency

GeneICL is highly efficient, requiring no gradient updates at inference and producing predictions within seconds on a laptop CPU The model size is 4.2M parameters, which is approximately 387× fewer than TabFM (1.65B), the strongest competing general-purpose tabular foundation model on regression When run on a consumer laptop CPU, one cross-validation fold takes about 7.5 s on average, with peak memory below 5 GB

Limitations

The authors note that their prior is derived from bulk RNA-seq, meaning single-cell RNA-seq and other omics modalities remain untested Furthermore, every task is evaluated within a single cohort, which leaves robustness to batch and platform shifts between support and query samples open They also consider whether the small margin over linear baselines reflects a limit of what tens to hundreds of expression profiles can reveal or a limit of current priors and architectures

Conclusion

The pretraining objective, rather than model scale, appears to be what limits current models Supervised pretraining benefits from tasks that resemble transcriptomic cohorts The realistic covariates matter more than fidelity to individual samples, as replacing the prior with the generic TabICLv2 prior lowers regression performance GeneICL's recurrent formulation supports substantially deeper iteration without destabilizing predictive performance, although additional recurrence does not provide a performance benefit on the present benchmark The evaluation suggests that GeneICL benefits more from generic variation in the representation or support set than from restricting inference to predefined transcriptomic feature subsets

How it works

1. GeneICL uses a two-stage encoder–ICL architecture, replacing full feature mixing with a row-attention bottleneck over [CLS] tokens The input is standardized and projected onto PCs fitted on the support set alone, reducing the number of feature tokens from G to at most ns-1

2. It employs a looped in-context learner where a single weight-tied block is applied recurrently over fixed evidence, separating fixed task evidence from an evolving hidden state The recurrence is defined by H(t+1) = fθ(H(t) + E), where E = [0; Zsupp; Zquery] denotes the fixed evidence and H(0) is initialized with learned thinking representations

3. For survival prediction, GeneICL-Cox regresses on null-model martingale residuals ri, which requires no model, and then regresses them with GeneICL in a single pass This allows the frozen model to predict survival by ranking patients by risk without architectural changes or survival-specific pretraining

Ablation Study Insights

The ablation study shows that replacing the single recurrent ICL block with eight untied blocks increases the parameter count from 4.24M to 19.25M but does not improve any of the evaluated metrics Replacing the bottleneck feature attention with full self-attention in the encoder increases complexity from O(n) to O(n 2) with matched or lower performance GeneICL's recurrent formulation remains stable far beyond the training depth, whereas plain recurrence degrades under continued iteration

Inference-Time Ensembling

GeneICL can benefit from inference-time ensembling by constructing multiple views of the same prediction problem using a single frozen GeneICL checkpoint Strategies include PCA-threshold ensembling, support-view ensembling, gene-subsample ensembling, rank/quantile ensembling, and class-permutation ensembling Generic strategies like PCA-threshold and supportview are generally competitive with the single-forward baseline and provide modest improvements for several metrics Transcriptomics-specific approaches based on highly variable genes or Hallmark pathways do not consistently improve performance

Benchmark Details

The benchmark comprises 27 classification and 25 regression tasks (52 in total) and separately, 28 survival tasks evaluated using SurvBoard Classification and regression tasks cover response to drugs, prognosis, and non-invasive diagnosis from blood transcriptomes Survival evaluation uses SurvBoard, which standardizes preprocessing and data splits for cancer survival models The median sample sizes across the three outcome types are 144 for classification, 491 for regression, and 202.5 for survival

Data Sources

The VAE is trained on human bulk RNA-seq profiles from ARCHS4, which spans diverse tissues, cell types, diseases, and experimental conditions The classification and regression tasks are drawn from large clinical cohorts like TARGET (Bolouri et al.

1. Bold Header: Transcriptomics-aware Pretraining Prior

GeneICL utilizes a semi-synthetic prior built from a variational autoencoder trained on real bulk RNA-seq data, sampling expression profiles from this VAE to generate synthetic datasets, which allows the model to learn the gene–gene correlation structure of real transcriptomes while retaining an effectively unlimited number of supervised tasks.

2. Bold Header: Training-Free Survival Prediction

The system enables survival prediction via a training-free reduction to regression using Cox partial-likelihood residuals, allowing the model, after being pretrained, to predict risk scores by regressing these residuals without requiring architectural changes or survival-specific pretraining.

3. Bold Header: Parameter Efficiency and Deployment Speed

GeneICL achieves high performance with significantly fewer resources, having up to 387× fewer parameters than TabFM and running on a consumer laptop CPU in seconds for inference, enabling deployment on standard hardware.

4. Bold Header: Robustness to Representation Choices

The model demonstrates stability across different representation choices; for instance, when given all principal components at inference instead of those explaining 90% of the support-set variance, performance remained largely unchanged in classification (AUROC 0.847 vs 0.846).