Evaluating utility of clinical prediction models: A comparison of decision curve analysis and cost curves

The gist The comparison between decision curve analysis and cost curves reveals that while both approaches assess model utility across different thresholds, they are closely related, with decision curves being closely related to a specific type of cost curve called a Brier curve <ref:2509.24608#pg20>.

Model Evaluation Approaches

Classification models typically predict a score and use a decision threshold to produce a classification <ref:2509.24608#pg2>. Appropriate model evaluation should carefully consider the context in which a model will be used, including the relative value of correct classifications of positive versus negative examples, which affects the threshold that should be used <ref:2509.24608#pg2>. Decision curve analysis (DCA) and cost curves are model evaluation approaches that assess the expected utility and expected loss of prediction models, respectively, across decision thresholds <ref:2509.24608#pg2>.

Relationship between DCA and Cost Curves

The paper demonstrates that decision curves are closely related to a specific type of cost curve called a Brier curve <ref:2509.24608#pg2>. Both curves are derived assuming model scores are calibrated and setting the classification threshold using the relative value of correct positive and negative classifications, and the x-axis of both curves are equivalent <ref:2509.24608#pg2>. Net benefit (used for DCA) and Brier loss (used for Brier curves) will always choose the same model as optimal at any given threshold <ref:2509.24608#pg2>. Across thresholds, differences in Brier loss are comparable whereas differences in net benefit cannot be compared <ref:2509.24608#pg2>.

Decision Curve Analysis (DCA)

DCA is a visual approach that was proposed in 2006 to assess the utility of a clinical prediction model <ref:2509.24608#pg2>. It uses the relative value of correct positive and negative classifications to choose relevant thresholds (as above) and estimate a ‘net benefit’ measure reflecting clinical utility <ref:2509.24608#pg2>. A prediction model is deemed to have clinical utility at relative classification values where the net benefit value is above both baselines <ref:2509.24608#pg2>. Net benefit is a measure of the expected utility of a prediction model, capturing the difference between the gain for correct predictions of the positive class and the loss for incorrect predictions of the negative class defined [18] as: NB(t) = UP/Pt n - UN/Fp n <ref:2509.24608#pg2>.

Cost Curves and Brier Curves

Cost curves were initially proposed in 2005, as a general approach for assessing classifier performance across a range of possible operating conditions (class distributions and misclassification costs) <ref:2509.24608#pg2>. The paper focuses on an alternative cost curve described by Hernandez-Orallo et al. [14] where the loss is defined as lossCP(t, C) = 2((1 − C)πP (1 − T P Rt) + CπN F P Rt) <ref:2509.24608#pg2>. A cost curve that uses the probabilistic threshold choice method is referred to as a Brier curve, and the loss at each cost proportion is given by BCCP(t) = 2(1 − t)πP (1 − T P Rt) + tπN F P Rt <ref:2509.24608#pg2>. The area under the Brier curve (with cost as the operating condition) is the Brier score which makes clear the relationship between Brier score and clinical utility <ref:2509.24608#pg2>.

Comparison of Metrics

The paper shows that Net benefit and Brier loss are linearly related for a given threshold choice such that, at a particular threshold, they will always prefer the same model over another <ref:2509.24608#pg2>. However, differences in net benefit across thresholds are not directly comparable as they correspond to different differences in expected utility <ref:2509.24608#pg2>. In contrast, differences in Brier loss are comparable across thresholds and can be used for comparison <ref:2509.24608#pg2>. The upper envelope decision curve is a useful addition to DCA plots as it shows, for a given model, how much clinical utility could be improved through recalibration alone <ref:2509.24608#pg2>.

Key Interpretations

The paper notes that a 0.01 higher net benefit is equivalent to 1 more TP per 100 examples or (1 − t)/t fewer FPs per 100 examples or something in between <ref:2509.24608#pg2>. A 0.01 higher Brier curve loss is equivalent to 5/t more FPs per 1,000 examples or 5/(1 − t) more FNs per 1,000 examples, or something in between <ref:2509.24608#pg2>. The Brier score combines refinement and calibration loss <ref:2509.24608#pg2>.

Conclusion on Applicability

Brier curves are more generally applicable (when a wider range of thresholds are plausible) <ref:2509.24608#pg2>. Decision curves allow the interpretation in terms of the number of true positives per n examples <ref:2509.24608#pg2>. The paper suggests that the upper envelope decision curve is a useful addition to indicate the possible gain in net benefit that could be achieved through recalibration alone <ref:2509.24608#pg2>. The authors conclude that Brier curves are more generally applicable across a wider range of decision thresholds and might be more easily interpreted across thresholds since a unit difference in Brier loss corresponds to the same difference in model value across thresholds <ref:2509.24608#pg2>.

Discussion on Utility Setting

A key difference between these approaches is the way class utilities/costs are set, as DCA sets the utility of classifying a positive example correctly to 1 across all thresholds, and the utility of classifying a negative example correctly varies across thresholds <ref:2509.24608#pg2>. The reason for the choice of utility values used in DCA is so that net benefit can be easily interpreted as an equivalent number of true positive results per n examples, assuming no change in false positive results <ref:2509.24608#pg2>. The authors believe that differences in net benefit values across thresholds may be misleading as a given increase in net benefit corresponds to different increases in expected utility across thresholds <ref:2509.24608#pg2>. Beneficial repercussions include a better range of possible net benefit values across t, with maxNB(t) = 2(1 − t)πP and minNB(t) = −2tπN, particularly that the minimum net benefit no longer approaches −∞ when t → 1 <ref:2509.24608#pg2>.

Comparison of Comparison Curves

The 'treat all' and 'treat none' lines common on DCA plots can also be shown in cost space, and we call them ‘predict all as positive’ and ‘predict all as negative’ to refer to the general case <ref:2509.24608#pg2>. The Brier curve is below these lines at the same threshold scores as the decision curve is above the treat all and treat none lines, such that the model would be deemed better than the two comparisons at the same threshold values <ref:2509.24608#pg2>. The upper envelope decision curve can be calculated, and compared with the model decision curve <ref:2509.24608#pg2>.

Final Summary

In summary, Brier curves are an alternative to decision curves that plot expected loss against threshold score <ref:2509.24608#pg2>. Our aim was not to suggest that one curve is overall better than the other but to present the properties of each so that practitioners can make an informed decision about which to use <ref:2509.24608#pg2>. Brier curves are more generally applicable across a wider range of decision thresholds and might be more easily interpreted across thresholds since a unit difference in Brier loss corresponds to the same difference in model value across thresholds <ref:2509.24608#pg2>. The area under the Brier curve is the Brier score which makes clear the relationship between Brier score and clinical utility <ref:2509.24608#pg2>. The decomposition into refinement and calibration loss can be seen by plotting the lower envelope cost curve in addition to the Brier curve <ref:2509.24608#pg2>.

Statement on Funding

The authors received no funds, grants, or other support during the preparation of this manuscript <ref:2509.24608#pg2>. The authors declare they have no competing interests <ref:2509.24608#pg2>.

1. Model selection can be optimized by comparing Net Benefit across multiple operating contexts using decision curves to see how models perform relative to clinical utility across different scenarios. This allows researchers to choose a model that maximizes expected utility based on the specific relative value of correct positive versus negative classifications.

2. Recalibration of model scores can be assessed for its potential gain in net benefit by calculating the upper envelope decision curve, which shows the possible gain in net benefit that could be achieved through recalibration alone. This helps determine if improving score calibration is a more effective way to enhance clinical utility than changing the predictive model directly.

3. Performance assessment across a wider range of thresholds can be done using Brier curves because they are more generally applicable (when a wider range of thresholds are plausible). This provides a consistent measure of expected cost across various operating conditions, which is useful when the specific decision threshold is not fixed beforehand.

Classification models typically predict a binary outcome and use one or more decision thresholds to produce a classification or inform a clinical decision. Decision curves and cost curves are model evaluation approaches that assess net benefit and loss across decision thresholds. In this paper we compare decision curves and cost curves, demonstrating that decision curves are closely related to a specific type of cost curve called a Brier curve. Users of both approaches set the classification threshold assuming calibrated model scores, using the relative clinical importance of correct positive and negative classifications. A given decision curve has a corresponding Brier curve such that reference lines common in each space can be included in either (e.g. the 'treat none' decision curve can be translated into an equivalent 'treat none' Brier curve). We suggest the upper envelope decision curve as a useful comparison to a model decision curve showing the maximum achievable net benefit through recalibration. We show that net benefit (used for decision curves) and Brier loss (used for Brier curves) will always choose the same model as optimal at any given threshold, but cannot be compared or aggregated across thresholds except under specific assumptions; doing so otherwise requires custom weightings.