PPG-Bench: A Unified Benchmark for PPG Foundation Models

arXiv:2610.13435 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "PPG-Bench: A Unified Benchmark for PPG Foundation Models".

Jane: The gist: Current PPG foundation models are not yet strong general-purpose representations because across many tasks, they do not provide a meaningful advantage over random chance / constant baselines,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to wrap up, we're talking about PPG-Bench: A Unified Benchmark for PPG Foundation Models by Arvasu Kulkarni Mannas and his team. This paper is essentially showing that the models we've been developing for photoplethysmography are not yet strong general-purpose representations.

Jane: They claim this because when you test them against the random/constant baseline, they don't show a meaningful advantage in thirty-five point three percent of the sixty eight task regime combinations <ref:2610.13435#pg2>. That’s a pretty low number for something that sounds so impressive at first glance.

Lu: What this implies is that we can't just assume these foundation models are good across all tasks without proving they beat simpler controls like pulse-rate-variability or MOMENT.

Meng: From a deployment view, this means we need to be very careful. If a model only beats something that’s already simple, it might not be worth the complexity of running it in production on real traffic.

Lalam: The authors are motivated by the fact that current PPG foundation models don't provide a meaningful advantage over random chance, simple physiological features, or a generic time-series foundation model.

Tom: So the implication is clear: future evaluation in this field needs to include checks against simple, non-learned controls to see if these models actually deliver value.

Jane: The whole point of PPG-Bench is to provide that common ground for comparison across different evaluations, ensuring that when we talk about foundation models for PPG, we are using a unified and rigorous standard.

Conclusion: Segment: Conclusion**

Tom: So, we’re wrapping up PPG-Bench by Arvasu Kulkarni Mannas and his team. The main thing they're saying is that current PPG foundation models aren't actually strong general tools yet because they don't consistently beat simple controls or random chance across all tasks.

Jane: So if I’m driving, what does that mean for me? It means if you’re trying to use an AI model to predict cardiovascular state from a heartbeat signal, you can’t just pick the most complex model; you have to check if it's actually better than just looking at the raw pulse rate variability or some basic time-series math.

Lu: Exactly. The authors found that for over a third of those sixty eight tests, the foundation model didn't give any real advantage over just guessing randomly or using a fixed baseline feature set like PRV or MOMENT.

Meng: From my side, it means deploying these models right now is risky because they aren't proven to be robust when things get messy at scale. If they only beat the simple stuff, you’re still stuck with all that heavy computation and maintenance overhead for no real gain.

Lalam: I’m pretty sure this paper confirms what I’ve been seeing in the literature: we need to judge these models on whether they provide a measurable gain over much simpler, non-learned alternatives. The results are quite definitive on that point.

Tom: It boils down to a check: if a model can’t outperform those basic controls, it probably isn't providing the kind of transferable representation we hoped for yet.

Jane: So the whole goal of this benchmark is to move past just showing impressive numbers and start asking if these models offer anything useful beyond just being technically complex. Next up, we’ll look at how they actually tested these models across nine different datasets.

Arvasu Kulkarni, Mahir Jain, Aditya Ray Mishra, Sandeep Singh, Siddharth Panwar

Mannas AI

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

Code: https://github.com/mannas-ai/ppg-bench

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 82/100

The gist: The gist: Current PPG foundation models are not yet strong general-purpose representations because across many tasks, they do not provide a meaningful advantage over random chance / constant

Key concepts

PPG Foundation Models (FMs): These are large AI models trained on Photoplethysmography (PPG) signals to learn reusable representations. Researchers developed them for various tasks, but their actual usefulness is hard to measure because different studies use different setups, making comparison difficult.
Downstream Regimes: This refers to four ways models are tested for adaptability: linear probing (testing direct feature access), random forests (testing nonlinear boundaries with a fixed encoder), frozen MLP probes, and end-to-end finetuning. These regimes test how much flexibility a model gains when adapting to different tasks.
Controls: These are simple baselines used to judge if the complex FM is actually useful. The controls include a random/constant baseline, deterministic pulse-rate-variability (PRV) features, and MOMENT, which is a generic time-series foundation model. If an FM doesn't beat these controls, it's not providing significant value.
Subject-Disjoint Cross-Validation: This rigorous testing method ensures that the performance of the models is evaluated fairly. It means data from one subject cannot be used to train or test another model, which prevents artificially inflated results and confirms the model's true generalizability.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes

Summary

The gist: Current PPG foundation models are not yet strong general-purpose representations because across many tasks, they do not provide a meaningful advantage over random chance / constant baselines, simple physiological features, or a generic time-series foundation model.

Introduction and Motivation

Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, leading to the development of PPG foundation models (FMs) intended to compute reusable representations for various downstream tasks Different evaluations use different datasets, task definitions, preprocessing, downstream heads, hyperparameter tuning, and baselines. The paper introduces PPG-Bench to address the difficulty in comparing these models by providing a unifying open-source benchmark evaluating five public PPG FMs across nine open-access datasets and 26 tasks spanning cardiovascular state estimation, activity, sleep, screening, and identification.

Benchmark Design

PPG-Bench is designed to test models under four downstream regimes of increasing adaptability: linear probing, random forests, frozen MLP probes, and end-to-end finetuning. The evaluation also includes three controls: random/constant baseline, a deterministic pulse-rate-variability (PRV) feature set, and a generic time-series FM (MOMENT). The benchmark employs model-specific hyperparameter optimization and subject-disjoint cross-validation to ensure rigorous testing.

Datasets and Tasks

The benchmark utilizes nine open-access datasets covering 17 primary and nine secondary tasks, spanning cardiovascular estimation, activity, affect, sleep, screening, and identification. Preprocessing is applied minimally to preserve each model’s native pipeline while converting source data into a common segmented representation. The researchers critically evaluated task inclusion by excluding targets that are externally available or low-incremental-value, and by defining ambiguous labels using established clinical thresholds rather than arbitrary binarization.

Evaluation Results

The overall results show that 35.3% of the 68 task–regime combinations show no meaningful FM advantage over the random/constant baseline. Furthermore, in 83.8% of cases, no PPG-specific FM outperforms both PRV and MOMENT. Near-chance performance is observed on several tasks, such as 4 of 17 tasks under linear probing. Simple physiological features and MOMENT are highly competitive, with PPG-specific FMs frequently failing to outperform the deterministic PRV control.

Analysis of Downstream Adaptation

The effect of increased downstream flexibility is not uniform; Random Forests generally underperform linear probing when FMs are better than chance, while frozen MLP probes often recover or improve upon linear performance. Finetuning further improves several regression tasks but is less consistent for classification. When performance is no better than the control, added flexibility can be harmful, as linear-to-finetuning produces a mean change of −29.9%. The benchmark also confirms that subject-level aggregation improves performance in every comparison, supporting the principle of evaluation at the same level at which the ground-truth label is defined.

Conclusion

Current PPG FMs are not yet consistently strong general-purpose representations because many tasks remain near chance or are matched by PRV or MOMENT. The results suggest that current PPG FMs do not provide a meaningful advantage over random chance / constant baselines, simple physiological features, or a generic time-series foundation model. This indicates that progress in the field should be judged by whether those models provide robust value over much simpler alternatives.

--- Page 1 ---

PPG-Bench: A Unified Benchmark for PPG Foundation Models Arvasu Kulkarni Mannas AI arvasu@mannas.ai Mahir Jain Mannas AI mahir@mannas.ai Aditya Ray Mishra Mannas AI aditya@mannas.ai Sandeep Singh Mannas AI sandeep@mannas.ai Siddharth Panwar Mannas AI siddharth@mannas.ai Abstract Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream tasks These models are difficult to compare: different evaluations use different datasets, task definitions, preprocessing, downstream heads, hyperparameter tuning, and baselines. We present PPG-Bench, a unifying open-source benchmark evaluating five public PPG FMs across nine open-access datasets and 26 tasks spanning cardiovascular state estimation, activity, sleep, screening, and identification. We evaluate models under four downstream regimes of increasing adaptability (linear probing, random forests, frozen MLP probes, and end-to-end finetuning), with model-specific hyperparameter optimization and subject-disjoint cross-validation. We also evaluate against three controls: random/constant baseline, a deterministic pulse-rate-variability (PRV) feature set, and a generic time-series FM (MOMENT). Across all 68 task–regime combinations, 35.3% show no meaningful FM advantage over the random/constant baseline, and in 83.8% no PPG-specific FM outperforms both PRV and MOMENT. We also find that added downstream head capacity generally improves performance, but can cause significant degradation if the model is performing at chance/constant level. Current PPG FMs therefore cannot yet claim to provide generalised and transferrable representations of PPG signals, and future evaluation in this field should include checks against simple, non-learned controls.

--- Page 2 ---

To address these issues, we introduce PPG-Bench, an open-source benchmark that tests five public PPG FMs across nine open-access datasets and 17 primary + 9 secondary tasks. Models are evaluated using linear probes, random forests, frozen MLP probes, and end-to-end finetuning, with modelspecific hyperparameter optimization and fully subject-disjoint cross-validation. We compare them against a random/constant baseline, a deterministic pulse-rate-variability (PRV) representation, and MOMENT, a generic time-series FM (Goswami et al., 2024). Across 68 task–regime combinations (regime meaning the type of evaluation method used), 24 (35.3%) show no meaningful PPG-FM improvement over the random chance baseline, and in 57 (83.8%) no PPG-specific FM outperforms both PRV and MOMENT. These results show that current PPG FMs provide inconsistent gains over simpler alternatives and motivate common, controlled evaluation.

--- Page 3 ---

We use nine open-access datasets covering 17 primary and nine secondary tasks (Appendix B.2 and Table 8) while varying in recording environment, hardware, and sensor location. Preprocessing PPG-Bench applies minimal preprocessing and otherwise preserves each model’s native preprocessing pipeline as faithfully as possible, to ensure released checkpoints do not encounter OOD (out-of-distribution) data. Source data are converted into a common segmented representation and resampled as required, while model-specific operations such as normalization, filtering, cropping, padding, or feature construction follow the preprocessing used by the corresponding pretrained model.

--- Page 4 ---

Four evaluation methods We evaluate each model under four downstream regimes with increasing flexibility. First, linear probing uses ridge or logistic regression on frozen embeddings, testing if task-relevant information is directly accessible (Alain and Bengio, 2016). Second, a Random Forest allows nonlinear decision boundaries while keeping the encoder fixed.

Improvements for AI systems

Bold Header: Generalization Assessment via Control Comparison

The improved AI system can robustly determine if a PPG foundation model provides meaningful FM advantage over simple, non-learned controls by explicitly testing against a random/constant baseline, a deterministic pulse-rate-variability (PRV) feature set, and a generic time-series FM (MOMENT).

Bold Header: Dynamic Downstream Regime Selection

The system can adapt its evaluation strategy based on the utility of the frozen representation by selecting between linear probing, random forests, frozen MLP probes, and end-to-end finetuning, as these regimes are evaluated under increasing adaptability.

Bold Header: Hyperparameter Optimization for Model Selection

The AI system can employ model-specific hyperparameter optimization by allowing models to select an optimization best suited to them using techniques like Optuna with Hyperband, addressing the sensitivity of rankings to training recipe choices.

Bold Header: Overfitting Mitigation via Capacity Control

When performance is low, the system can mitigate significant degradation by ensuring that added downstream head capacity does not lead to overfitting; this is achieved by monitoring transitions where added downstream head capacity generally improves performance, but can cause significant degradation if the model is performing at chance/constant level.

Bold Header: Subject-Level Evaluation for Trait Prediction

For diagnostic or trait-level labels, the system should employ a specific aggregation strategy—such as mean, median, and top-k mean—to determine the subject-level score rather than averaging it away with ambiguous windows, ensuring evaluation aligns with where the ground-truth label is defined.

Abstract

Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream tasks. These models are difficult to compare: different evaluations use different datasets, task definitions, preprocessing, downstream heads, hyperparameter tuning, and baselines. We present PPG-Bench, a unifying open-source benchmark evaluating five public PPG FMs across nine open-access datasets and 26 tasks spanning cardiovascular state estimation, activity, sleep, screening, and identification. We evaluate models under four downstream regimes of increasing adaptability (linear probing, random forests, frozen MLP probes, and end-to-end finetuning), with model-specific hyperparameter optimization and subject-disjoint cross-validation. We also evaluate against three controls: random/constant baseline, a deterministic pulse-rate-variability (PRV) feature set, and a generic time-series FM (MOMENT). Across all 68 task-regime combinations, 35.3% show no meaningful FM advantage over the random/constant baseline, and in 83.8% no PPG-specific FM outperforms both PRV and MOMENT. We also find that added downstream head capacity generally improves performance, but can cause significant degradation if the model is performing at chance/constant level. Current PPG FMs therefore cannot yet claim to provide generalised and transferrable representations of PPG signals, and future evaluation in this field should include checks against simple, non-learned controls. Code for the benchmark and instructions to evaluate more models can be found here - https://github.com/mannas-ai/ppg-bench.