Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets

arXiv:2601.14663 · eess.SY, cs.AI, cs.SY · Submitted 2026-01-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets".

Dev: Here's the gist of the paper. The gist The proposed framework integrates Monte Carlo dropout (MCD) with conformal prediction (CP) to produce calibrated, finite-sample prediction intervals for aggregated prosumer flexibility Reliable forecasting of prosumer flexibility is critical for demand response aggregators participating in frequency controlled ancillary service markets.

Rosa: Limited historical data, dependence on exogenous forecasts, and heterogeneous prosumer behaviour introduce significant epistemic uncertainty, making deterministic or poorly calibrated probabilistic models unsuitable for market bidding.

Dev: The Danish system operator has adopted a P90 compliance standard for frequency controlled ancillary services (FCAS) capacity market participants, allowing a capacity deviation with a maximum probability of 10%.

Rosa: Consequently, this study aims to develop a data-driven framework that captures epistemic uncertainty while providing formal coverage guarantees.

Paper summary: Rosa: So we're looking at this paper today titled "Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets." Basically, the big problem they're tackling is that reliably forecasting how flexible a bunch of prosumers will be is super critical for demand response aggregators.

Dev: Right, and these aggregators are playing in frequency controlled ancillary services markets where they have really strict reliability requirements, like the P90 standard which means you need to be reliable ninety percent of the time.

Taro: So they're dealing with a lot of uncertainty, like having limited historical data and depending on outside factors, plus prosumers are all acting differently. That makes using simple deterministic or poorly calibrated probabilistic models just not work for making bids in the market.

Rosa: Exactly, and that uncertainty is what makes them worried about getting it wrong when they're trying to participate in these markets. The paper proposes a scalable uncertainty quantification framework that combines monte carlo dropout, or MCD, with conformal prediction, or CP.

Dev: So the core idea is to use this MCD with CP combo to generate prediction intervals for the aggregated prosumer flexibility that are properly calibrated and based on a finite sample size.

Taro: Why is that combination necessary? I mean MCD is computationally efficient compared to bayesian neural networks, but it can sometimes give you uncertainty estimates that aren't quite right, especially when you have multiple outputs or long forecasting periods.

Rosa: That's the thing they point out, and then they integrate CP because CP gives them a lightweight way to get prediction intervals with finite-sample coverage guarantees under data exchangeability.

Dev: And this integration is supposed to make the uncertainty reliability better while still keeping the whole process tractable for real-world use.

Taro: But what about the actual application? The authors apply this framework to a behind-the-meter aggregator that's participating in the Danish manual frequency restoration reserve capacity market.

Rosa: That's a concrete example, and they use a large-scale synthetic dataset created from a modified home energy management system algorithm from Watts A/S, mixed with public data for loads, solar generation, prices, activations and device levels.

Dev: So this surrogate model they built is supposed to capture the aggregate prosumer price responsiveness while giving these uncertainty-aware estimates that are actually useful for making bids in the market.

Taro: It sounds like they're trying to figure out how to make a system that can handle real-world misbehavior and still give you a safe estimate when things go off script.

Rosa: The results show that if you just use MCD by itself, it systematically overestimates the available flexibility and ends up violating the P90 compliance standard they're aiming for.

Dev: But when they use their proposed MCD-CP framework, it achieves reliable coverage with controlled conservatism, which is important because you don't want to be too optimistic about what you can actually deliver.

Paper summary: Taro: So when this framework gets embedded into an aggregator bidding model, it substantially reduces the risk of overbidding and they manage to get up to seventy percent of perfect-information profit while still meeting those regulatory reliability constraints.

Rosa: It sounds like a very practical solution for forecasting flexibility under uncertainty in these specific types of markets.

Dev: So what does this mean for the broader context? The paper explores how these tools can give operators a way to make safer, more informed decisions when they are relying on predictions about how much flexibility they can actually get from prosumers.

Taro: It suggests that combining MCD with CP is a solid way to handle the epistemic uncertainty inherent in this kind of forecasting problem without getting bogged down in overly complex methods like full bayesian models.

Rosa: And it gives a clear path forward for how aggregators can forecast their flexibility more reliably and safely when they have to compete for capacity in these ancillary service markets.

Dev: So the implication is that we can move from models that are just potentially wrong to models that give us calibrated prediction intervals, which directly impacts how much risk an aggregator takes on its bids.

Taro: For someone just listening, this means thinking about how much wiggle room you need in your planning when you're trying to promise flexibility based on a forecast.

Rosa: And the authors found that among the different uncertainty quantification methods they tested—like standalone MCD, or MCP, or PCP—the PCP method produced the lowest miscoverage at zero point one four percent, confirming its strong reliability compared to others.

Dev: So while the full MCD-CP framework is shown to be effective, they also found that some other conformal prediction strategies like PCP and CCP are even more conservative but still preserve a higher share of profit after adjustment when compared to the overbidding prone benchmarks.

Taro: That suggests there's a trade-off between being very safe and making the most money in this context.

Rosa: So, to wrap up on the paper "Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets," they show that the hybrid MCD-CP approach delivers prediction intervals that satisfy P90 reliability while remaining economically viable in the Danish FCAS capacity market.

Dev: It's a method designed to control overbidding risk by giving aggregators better, more reliable forecasts of prosumer behavior, and it does this using techniques that are computationally efficient enough for real-time operational use.

Taro: The paper shows that when you combine the computational efficiency of MCD with the formal coverage guarantees from conformal prediction, you get a solution that works well in practice for these complex energy forecasting tasks.

Rosa: That's what they show us—a way to quantify uncertainty in prosumer flexibility that respects both the technical need for reliability and the economic need to make money.

Dev: So it's about building a tool that gives aggregators the confidence needed to participate in these markets without taking on unacceptable levels of risk.

Conclusion: Rosa: So, we're wrapping up this look at "Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets."

Dev: It really boils down to taking messy predictions about how flexible prosumers will be and giving you a range where you actually know the chance of being right.

Taro: The authors are using monte carlo dropout combined with conformal prediction for that, which is a way to get those reliable boundaries without needing a huge amount of historical data upfront.

Rosa: They tested this on some synthetic data from a Danish market, specifically for frequency controlled ancillary services where you need P90 compliance.

Dev: And the main point is that standalone methods were failing because they were just too optimistic, systematically overestimating the available flexibility and violating those reliability rules.

Taro: The MCD-CP framework fixed that by giving them a controlled conservatism, which means their prediction intervals actually cover what happens more often than they say do.

Rosa: When you plug this into an actual aggregator's bidding model, it cuts down on overbidding risk significantly, even getting close to perfect information profit while still hitting the regulatory constraints.

Dev: It’s a practical tool for a real-time loop because it gives you a safety case based on calibrated uncertainty instead of just one single guess.

Taro: The paper shows that while some other conformal prediction strategies are even more conservative, they still keep more profit after adjustment compared to the methods that were just overconfident.

Rosa: So, this framework gives operators a way to forecast flexibility with confidence and control the risk they take when bidding in these energy markets.

Dev: It shows that combining computational efficiency with formal coverage guarantees is a viable path for handling uncertainty in real-world energy systems.

Taro: But we still have questions about how well this framework holds up when the real world gets weird—when the prosumers’ behavior totally misbehaves outside of the synthetic data they used to train it.

School of Electrical and Mechanical Engineering, The University of Adelaide · Watts A/S

eess.SY, cs.AI, cs.SY

Submitted: 2026-01-21

Updated: 2026-10-22

Comments: Single column 31 pages, 10 figures, 3 tables, submitted for review to Applied Energy

Journal ref: Appl. Energy 427 (2026) 129059

DOI: 10.1016/j.apenergy.2026.129059

Code: https://github.com/Rose-STL-Lab/CopulaCPTS

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The proposed framework integrates Monte Carlo dropout (MCD) with conformal prediction (CP) to produce calibrated, finite-sample prediction intervals for aggregated prosumer flexibility

Key concepts

Monte Carlo dropout (MCD): MCD is a technique used to capture epistemic uncertainty in machine learning models without the complexity of Bayesian neural networks. It generates a predictive distribution that approximates a deep Gaussian process, allowing the model to quantify how uncertain it is about its predictions based on limited data.
Conformal Prediction (CP): CP provides a lightweight, distribution-free method to generate prediction intervals with formal coverage guarantees. It ensures that the calculated uncertainty regions reliably cover the true value with a specified probability, making it ideal for providing reliable bounds in forecasting scenarios.
Prosumer Flexibility Aggregation: This refers to combining the variable energy contributions of individual prosumers (those who both consume and produce energy) into a single, predictable pool. Forecasting this aggregated flexibility is essential for aggregators participating in markets that require precise control over frequency response.
P90 Standard Compliance: The P90 standard dictates a strict reliability requirement in ancillary service markets, meaning the forecast must be accurate at least 90% of the time. This study focuses on ensuring the proposed forecasting method meets this specific regulatory threshold for capacity market participation.

Terminology

Uncertainty Quantification — 2× in this paper · explained in 9 episodes
Conformal Prediction — 1× in this paper · explained in 11 episodes
Epistemic Uncertainty — 1× in this paper · explained in 12 episodes
Monte Carlo Dropout — 1× in this paper · explained in 3 episodes
Surrogate Model — 1× in this paper · explained in 3 episodes

Summary

The gist The proposed framework integrates Monte Carlo dropout (MCD) with conformal prediction (CP) to produce calibrated, finite-sample prediction intervals for aggregated prosumer flexibility

Motivation and Problem

Reliable forecasting of prosumer flexibility is critical for demand response aggregators participating in frequency controlled ancillary service markets. Limited historical data, dependence on exogenous forecasts, and heterogeneous prosumer behaviour introduce significant epistemic uncertainty, making deterministic or poorly calibrated probabilistic models unsuitable for market bidding. The Danish system operator has adopted a P90 compliance standard for frequency controlled ancillary services (FCAS) capacity market participants, allowing a capacity deviation with a maximum probability of 10%. Consequently, this study aims to develop a data-driven framework that captures epistemic uncertainty while providing formal coverage guarantees.

Methodology

The proposed framework is based on a hybrid MCD and CP method to capture uncertainty in the behaviour of prosumers, which is denoted as the prediction algorithm Φ1−α(·).

** Monte Carlo dropout (MCD) is used as a scalable and computationally efficient alternative to Bayesian neural networks (BNNs), effectively capturing epistemic uncertainty. It produces a predictive distribution that can be interpreted as an approximation of a deep Gaussian process. **

** Conformal prediction (CP) provides a lightweight and distribution-free framework to obtain prediction intervals with finite-sample coverage guarantees under data exchangeability. The integration of MCD with CP enhances uncertainty reliability while retaining tractability. **

The prediction region ξ is defined using Algorithm 1, which computes the adjusted quantile level qˆ to achieve a target coverage level of 1 − α. The bounding hyperrectangle Bξ encloses the conformal prediction region ξ, and this conservative bound is advantageous for downstream optimisation problems involving joint chance constraints.

Case Study and Data Generation

A large-scale synthetic dataset was generated using a modified industry-grade home energy management system (HEMS) algorithm from Watts A/S, combined with publicly available datasets for load and solar profiles, market prices, activation data, and device-level size distributions. The resulting machine learning surrogate model captures aggregate prosumer price responsiveness and provides uncertainty-aware flexibility estimates suitable for market bidding. The dataset D is exchangeable, as each scenario depends only on its own inputs, and is used exclusively to train and validate the proposed ML surrogate model Φ1−α(·) following the training procedure described in Section 4.2.

Results and Performance

The study compares three baseline UQ methods based on NN with MCD outputs, individual quantiles, naïve joint quantiles, MCP (MCD–CP), MMCP (MCD–CP), PCP (Probabilistic CP), and CCP (Copula CP).

** The results show that standalone MCD systematically overestimates available flexibility and violates P90 compliance, whereas the proposed MCD–CP framework achieves reliable coverage with controlled conservatism. **

** When embedded in an aggregator bidding model, conformalised methods substantially reduce overbidding risk and achieve up to 70% of perfect-information profit while satisfying regulatory reliability constraints. **

The CP-based methods meet their intended coverage, while the nonCP baselines show near-zero joint PICP, reflecting overconfident intervals. Among these, PCP produces the lowest miscoverage (0.14%), confirming its strong reliability.

Conclusion

The hybrid MCD-CP-based methods deliver well-calibrated prediction intervals and effectively control overbidding risk in capacity markets. The results show that the hybrid MCD–CP framework satisfies the P90 reliability requirement while remaining economically viable in the Danish FCAS capacity market. The proposed approach was evaluated in terms of both P90 compliance and economic performance. While PCP and CCP are more conservative, they preserve a higher share of PI profit after adjustment compared to the overbidding-prone benchmarks. The findings suggest that complementary policy or market design measures could be considered to support and potentially enhance prosumer engagement.

Improvements for AI systems

Scalable Uncertainty Quantification Framework: The system can generate calibrated, finite-sample prediction intervals for aggregated prosumer flexibility by integrating monte carlo dropout (MCD) with conformal prediction (CP). This framework addresses significant epistemic uncertainty arising from limited training data and dependence on exogenous forecasts.
Market Bidding Optimization Under Uncertainty: The system can solve a decision-dependent chance-constrained optimization problem to determine the optimal bid and incentive price simultaneously. This allows the aggregator to maximize profit while satisfying regulatory constraints, such as Energinet's P90 reliability requirement.
Risk Mitigation in Market Entry: By using this hybrid MCD–CP framework, the system can substantially reduce overbidding risk and achieve up to 70% of perfect-information profit while satisfying regulatory reliability constraints.

Abstract

Reliable forecasting of prosumer flexibility is critical for demand response aggregators participating in frequency controlled ancillary services market, where strict reliability requirements such as the P90 standard are enforced. Limited historical data, dependence on exogeneous factors, and heterogenous prosumer behaviour introduce significant epistemic uncertainty, making deterministic or poorly calibrated probabilistic models unsuitable for market bidding. This paper proposes the use of scalable uncertainty quantification framework that integrates Monte Carlo dropout (MCD) with conformal prediction (CP) to produce calibrated, finite sample prediction intervals for aggregated prosumer flexibility. The proposed framework is applied to a behind-the-meter aggregator participating in the Danish manual frequency restoration reserve capacity market. A large-scale synthetic dataset is generated using a modified industry-grade home energy management system, combined with publicly available load, solar, price, activation and device-level data. The resulting machine learning surrogate model captures aggregate prosumer price responsiveness and provides uncertainty-aware estimates suitable for market bidding. Multiple multivariate CP strategies are evaluated and benchmarked against conventional MCD-based methods. Results show that standalone MCD systematically overestimates available flexibility and violates P90 compliance, whereas the proposed MCD-CP framework achieves reliable coverage with controlled conservatism. When embedded in aggregator bidding model, conformalised methods substantially reduce overbidding risk and achieve upto 70% of perfect-information profit while satisfying regulatory reliability constraints, providing practical, computationally efficient, and market-compliant solution for aggregator flexibility forecasting under uncertainty.