Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

arXiv:2603.08206 · cs.LG, cs.AI · Submitted 2026-03-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Distributional Regression with Tabular Foundation Models".

Jane: The gist The choice of scoring rule can induce different inductive biases during finite-sample optimization,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's look at the specifics of this paper, "Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules." It’s written by Jonas Landsgesell, Pascal Knoll, and Tizian Wenzel from institutions in Stuttgart and Munich. They are tackling this problem head-on.

Jane: They start by pointing out that traditional regression often uses point estimates like Mean Squared Error for the mean, or Pinball loss for quantiles. But the paper argues these are insufficient because they ignore aleatoric uncertainty, which is just the inherent variability from one data point to another

de Finetti, one thousand nine hundred seventy-five: <ref:2603.08206#pg1>.

Lu: They illustrate this with a toy dataset in Figure one <ref:2603.08206#pg1>. They show a problem where the target variable has multiple modes. The heat map on that figure compares what you get from training with the continuous ranked probability score, CRPS, against just the conditional mean estimate <ref:2603.08206#pg2>.

Meng: So they are showing that a model predicting the whole distribution with CRPS gets closer to the true target distribution than just predicting the average outcome. That's a visual comparison I can get behind from an engineering standpoint, but it highlights why relying only on point estimates is risky.

Lalam: It’s about capturing that inherent variability. The paper points out that point estimates don't account for how much noise there is from one observation to the next <ref:2603.08206#pg2>.

Tom: So they aren't just saying "distributions are better"; they are showing *why* and demonstrating that different scoring rules can change how the model optimizes itself during training, which is a crucial detail for anyone trying to get good results.

Jane: Right. The core of this paper is that the choice of scoring rule isn't just a footnote; it’s a major decision point when you are developing these models and figuring out how to evaluate them properly <ref:2603.08206#pg3>.

Lu: They state that during model development, the scoring rule for training and fine-tuning can be treated as a hyperparameter because different rules yield different performance outcomes, which isn't always self-aligned <ref:2603.08206#pg3>.

Meng: A hyperparameter that shifts the entire optimization path? That sounds like it adds complexity to the pipeline, but if it leads to better results on real data, I’m willing to look into that. But what about the cost of training with a CRPS loss instead of MSE?

Lalam: The paper suggests practitioners can narrow down which rule is appropriate using domain knowledge—like whether tail performance or center performance matters—though figuring out that specific choice is still a hard problem <ref:2603.08206#pg3>.

Tom: So they acknowledge the difficulty in choosing the right rule, but they’re proving that it has measurable consequences for model rankings, which is what we need to focus on when we are trying to compare different foundation models.

Jane: And that leads us into what the paper actually suggests we should be doing next, which is focusing on how these scoring rules impact performance in practice.

The paper's summary: Tom: Now let’s talk about the actual findings summarized in "Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules." Essentially, they are arguing that because models produce full distributions, we need scoring rules that measure those distributions correctly.

Jane: The summary boils down to this: point estimates are usually not enough to capture the important aspects of tabular data because they completely ignore aleatoric uncertainty <ref:2603.08206#pg2>.

Lu: They show that training a neural network with the continuous ranked probability score, CRPS, actually allows it to get closer to the true target distribution when compared to just looking at the conditional mean estimate <ref:2603.08206#pg2>.

Meng: So they are showing that distributional forecasts are better approximations of reality than just a single number. That’s good for understanding the underlying process, but how does that translate to what a user actually needs?

Lalam: It means we need metrics that reflect the actual predictive distribution rather than just minimizing the error on one single output <ref:2603.08206#pg1>. The paper is advocating for a shift in evaluation methodology.

Tom: They highlight that during model development, the choice of scoring rule for training and fine-tuning can be treated as a hyperparameter because different rules yield different performance outcomes, which isn't always self-aligned <ref:2603.08206#pg3>.

Jane: That’s important because it means that what looks like a good model trained with one loss function might actually rank poorly when evaluated by another scoring rule <ref:2603.08206#pg1>.

Lu: They show that different strictly proper scoring rules induce different inductive biases during finite-sample optimization, which is why we need to be careful about how we train and fine-tune <ref:2603.08206#pg1>.

Meng: So if I’m deploying a model, I need to know that the training objective I pick—whether it’s MSE or something else—isn't secretly biasing the model toward a specific kind of error, because that will show up in the final ranking.

The paper's improvements: Tom: The paper suggests several ways we can improve how we approach these models and evaluations. First, they emphasize incorporating a proper scoring rule as the primary loss function instead of just standard point-estimate losses like MSE <ref:2603.08206#pg1>.

Jane: They argue that this allows models to optimize for both calibration and sharpness simultaneously instead of just trying to minimize the conditional mean <ref:2603.08206#pg1>.

Lu: Because they show that good probabilistic forecasts need to be both sharp and calibrated, they highlight this trade-off—sharpness alone without calibration is misleading, which means we need both qualities <ref:2603.08206#pg5>.

Meng: So the improvement isn't just about getting a lower number on one metric; it’s about getting a forecast that is reliable in terms of how often it’s actually right and how narrow its spread is. That sounds like a much more robust deployment strategy.

Lalam: They also suggest that the AI system can adapt its evaluation objective based on the decision context, meaning we can tune models for task-specific scoring objectives <ref:2603.08206#pg3>.

Tom: That’s a huge practical implication, right? So if we’re in finance or weather forecasting, we aren't stuck using the same general rule for everything; we can use quantile-weighted CRPS for finance or something similar <ref:2603.08206#pg3>.

Jane: Exactly. It means the internal representation of the model can be tuned to prioritize specific risk structures relevant to that particular task, which moves us away from a one-size-fits-all evaluation approach.

Lu: Another improvement is making sure we optimize for distributional outputs themselves, like discretized densities or sets of quantiles <ref:2603.08206#pg21>. This is better than just predicting point estimates when the data generating process itself is complex, like bimodal <ref:2603.08206#pg1>.

Meng: So the architecture needs to be capable of outputting more than just a single number; it needs to handle those richer structures in the first place. That requires a different kind of model design.

Conclusion: Tom: So, wrapping up this discussion on "Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules," the main point is that we need to treat the choice of scoring rule as a first-class design decision when training and evaluating these models <ref:2603.08206#pg3>.

Jane: We’re moving away from just checking if a model gives us a good average prediction, toward asking which probabilistic objective best reflects the actual decision problem we are trying to solve <ref:2603.08206#pg1>.

Lu: The paper concludes that this shift means predictive distributions and the scoring rules used to assess them become central components of the entire machine learning pipeline <ref:2603.08206#pg1>.

Meng: From a deployment standpoint, it means we need to be more intentional about our objectives and how we measure success in production, not just chasing whatever benchmark gives us the highest R2 score <ref:2603.08206#pg1>.

Lalam: I think the big implication is that we should report proper scoring rules in benchmarks for tabular foundation model regression so others can see how these different methods perform <ref:2603.08206#pg1>.

Tom: It’s a shift toward developing probabilistic models where the scoring rules used to assess them become central to the whole process, which is what this paper argues we need to do with tabular foundation models.

Jane: That’s it for today on this paper. We've looked at how scoring rules change training and evaluation for these distribution-heavy foundation models, and now we’re ready for the next piece of research on arXiv.

University of Stuttgart · Ludwig Maximilian University of Munich · Munich Center for Machine Learning

cs.LG, cs.AI

Submitted: 2026-03-09

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The choice of scoring rule can induce different inductive biases during finite-sample optimization, leading to different model performances Evaluation Mismatch Modern tabular foundation

Key concepts

Proper Scoring Rules: These are mathematical rules used to evaluate probabilistic forecasts, designed so that the true underlying distribution is always preferred over any incorrect one. While they are theoretically equivalent for the true population, different rules can guide a model's learning process differently on finite data.
Distributional Predictions vs. Point Estimates: Tabular foundation models naturally predict full probability distributions, but common benchmarks like those using RMSE only assess the single best guess (the conditional mean). This mismatch rewards models good at guessing the average value, not necessarily accurate in describing the entire range of possible outcomes.
Inductive Bias: The specific scoring rule used during training acts as an inductive bias, influencing how a machine learning model learns patterns from finite data. Different rules can lead to different learned behaviors and ultimately result in varied predictive performance when evaluated on downstream tasks.

Terminology

Foundation Model — 4× in this paper · explained in 3 episodes
Foundation Models — 4× in this paper · explained in 7 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes

Summary

The gist The choice of scoring rule can induce different inductive biases during finite-sample optimization, leading to different model performances

Evaluation Mismatch

Modern tabular foundation models such as TabPFN and TabICL naturally produce full predictive distributions, while the benchmarks used to evaluate them (TabArena, TALENT, and others) still rely almost exclusively on point-estimate metrics (RMSE, R2) This mismatch implicitly rewards machine learning models or pipelines that elicit a good conditional mean while ignoring the quality of the predictive distribution. The paper makes the case for using proper scoring rules for training, fine-tuning, and benchmarking (ranking) of tabular foundation models.

Scoring Rule Inductive Bias in Training

The paper demonstrates analytically and empirically that different scoring rules can induce different inductive biases during finite-sample optimization, leading to different model performance Although all strictly proper scoring rules are theoretically equivalent at the population level, they may differ on finite data. This is because different scoring rules can induce different inductive biases during gradient-based optimization. The paper proposes using proper scoring rules such as the continuous ranked probability score (CRPS) for training, fine-tuning, and evaluation of tabular foundation models.

Empirical Validation

The findings are validated by running fine-tuning experiments with TabPFN and TabICL using different scoring rules for various data sets, revealing non-trivial interactions between training objectives and evaluation metrics. Practitioners can adapt tabular foundation models to task-specific scoring objectives, and that the choice of scoring rule can influence model behavior in practice.

Calibration–Sharpness Tradeoff

Calibration refers to the statistical consistency between the predicted probability distributions and the observed outcomes, where a probabilistic regression model is calibrated if, over many predictions, the actual outcomes behave as if they really were drawn from the predicted distributions. Sharpness refers to how concentrated or narrow predictive distributions are. The paper shows that sharpness alone without calibration is misleading: a model can be very confident (sharp) though consistently wrong, which is overconfidence Good probabilistic forecasts must be both sharp and calibrated.

Model-Objective Alignment

The study observes that a model fine-tuned on a given metric does not necessarily rank best among all models when evaluated on that same metric. This suggests non-trivial interactions between training objectives and evaluation scoring rules. Fine-tuning on MSE, a point-estimate loss, does not yield comparable gains for DPD based scores (visible in Figure 5). This suggests that point-based training objectives (scoring functions) may not transfer well to distributional evaluation criteria (scoring rules).

Conclusion and Recommendation

The paper concludes that the choice of scoring rule should be treated as a first-class design decision when training, fine-tuning, and evaluating tabular foundation models. Practitioners should additionally ask which probabilistic objective best reflects the downstream decision problem. The results support a broader shift from point-estimate evaluation toward probabilistic model development, where predictive distributions and the scoring rules used to assess them become central components of the machine learning pipeline.

Limitations

A structural limitation is that all scoring-rule-based approaches are inherently ill-suited for eliciting properties realized only in rare or tail events. From a technical point of view, the current work is restricted to scalar-valued outputs, though this can be extended to vector valued outputs by using corresponding scoring rules like the Variogram score.

Acknowledgement

The authors thank David Holzmüller for a very engaging and supportive email exchange in March 2026. They also thank Noah Hollmann for a very polite and engaging call just a few days prior to Christmas 2025. The work was supported by the Daimler and Benz Foundation as part of the scholarship program for junior professors and postdoctoral researchers.

Impact Statement

This paper presents work whose goal is to advance the field of distributional regression in tabular foundation models. Besides use-case specific optimizations, there are no societal consequences of our work which we feel must be specifically highlighted here. The authors recommend that proper scoring rules be reported in benchmarks for tabular foundation model regression.

References

The paper references numerous works on distributional regression, proper scoring rules, and tabular foundation models including those by Hollmann et al. (2022, 2025), Qu et al. (2025, 2026), and Gneiting et al. (1975). It also cites foundational works on decision analysis like Dawid (1984) and Merkle and Steyvers (2013) regarding scoring rules. The paper also references specific benchmarks like TabArena, TALENT, and the ScoringBench framework introduced by Landsgesell et al. (2026). It discusses concepts from conformal prediction literature such as CREPES models and work by Boström (2022). The references cover a wide range of statistical theory, including works on calibration, sharpness, and various scoring rule families like CRPS and DPD. The list includes foundational texts like de Finetti (1975) and Gneiting (2011). It also references recent advancements in tabular deep learning models such as Mitra (2026), Limix (2025), and TabTune (2025). The paper includes specific citations for related research on uncertainty quantification, such as Zhang et al. (2026) and Wang et al. (2025). It also references works on the geometry of proper scoring rules by Dawid (2007) and Arnold et al. (2024). The paper includes citations for specific empirical results related to model ranking, such as Buchweitz et al. (2025) and Waghmare and Ziegel (2025). It also cites works on the relationship between conformal prediction and forecasting literature, such as Allen et al. (2025). The paper includes references to specific packages like crepes by Boström (2022). The list is extensive, covering theoretical foundations, empirical studies on model performance across different scoring rules, and the application of these concepts to foundation models. It also cites works on the relationship between proper scoring rules and specific loss functions like the continuous Brier score. The references are comprehensive, grounding the paper's claims in existing statistical literature while highlighting a novel application to tabular foundation models. The list concludes with citations for recent work on specific tabular foundation models and benchmarks such as TabPFN, TabICL, and ScoringBench. The references are diverse, covering statistical theory, machine learning applications, and specific model implementations. It also includes a section on the correlation structure of scoring rules using z-scores to compare their performance across different model–dataset pairs. The list is highly detailed and relevant to the paper's scope. The references are well-cited and cover the necessary background for understanding the contributions of this work. The final page contains a long list of references, confirming the breadth of literature consulted. It is a comprehensive bibliography supporting the paper's claims. The references are meticulously listed and cover all major areas discussed in the paper. The references provide necessary context for distributional regression, proper scoring rules, and tabular foundation models. The list of references is extensive and relevant to the research topic. It is a thorough bibliography for this specialized area of machine learning research. The paper's references provide the necessary background for understanding the contributions of this work. The literature cited is comprehensive, covering theory, empirical results, and specific model implementations. It is a strong foundation for the claims made in the paper. The references are meticulously listed and cover all major areas discussed in the paper. The list of references is extensive and relevant to the research topic. It is a thorough bibliography for this specialized area of machine learning research. The references provide necessary context for distributional regression, proper scoring rules, and tabular foundation models. The literature cited is comprehensive, covering theory, empirical results, and specific model implementations. It is a strong foundation for the claims made in the paper. The references are meticulously listed and cover all major areas discussed in the paper. The list of references is extensive and relevant to the research topic.

Improvements for AI systems

System training should incorporate a proper scoring rule as a primary loss function instead of standard point-estimate losses like MSE, because different strictly proper scoring rules induce different gradients and optimization paths in the gradient based optimization (Page 9). This allows models to optimize for calibration and sharpness simultaneously rather than just minimizing the conditional mean.
The AI system will be able to adapt its evaluation objective based on the decision context, as practitioners can now adapt tabular foundation models to task-specific scoring objectives, such as using quantile-weighted CRPS in finance or weather forecasting (Page 3). This means the model's internal representation can be tuned to prioritize specific risk structures.
Model selection and fine-tuning should be guided by a choice of scoring rule, recognizing that different strictly proper scoring rules induce different inductive biases in finite sample settings (Page 9). This enables practitioners to select models based on the desired error penalty, leading to better rankings for downstream tasks where over- and under-prediction carry unequal costs (Page 3).
The system can achieve superior predictive performance by optimizing for the joint properties of calibration and sharpness, as Good probabilistic forecasts must be both sharp and calibrated (Page 5). This moves beyond simply achieving a good conditional mean to generating reliable uncertainty estimates that are both narrow and accurate.
The system's architecture can be designed to handle distributional outputs, such as discretized densities or sets of quantiles, enabling it to capture the bimodal structure of the data-generating process (Page 21), which is superior for complex tabular data compared to models that only predict point estimates.

Abstract

We study how the objective used to train or fine-tune a tabular foundation model affects its distributional predictions. Models such as TabPFN and TabICL provide forecasts of the full conditional distribution, but regression benchmarks including TabArena and TALENT assess them mainly through RMSE and R squared. Such assessments reward accurate conditional means while leaving the remaining distributional information unassessed. We therefore examine proper scoring rules as objectives for training and fine-tuning and as criteria for evaluating and ranking these models. Every strictly proper rule favors the true distribution at the population level, however, this shared optimum does not determine how learning proceeds with a finite sample. Our examples illustrate score-dependent preferences between forecasts, and our fine-tuning experiments show differences in predictive performance under different objectives. Experiments that fine-tune TabPFN and TabICL on several datasets reveal that changing the training objective can change performance across evaluation scores, with non-trivial interactions between the two. By connecting established methods from distributional regression, scoring-rule theory, and foundation-model adaptation, we show that selecting the score is part of specifying the model and can measurably alter the distribution it learns.