How Should Transformers Encode Numeric Values in Electronic Health Records?

arXiv:2607.01391 · cs.LG, cs.AI · Submitted 2026-07-01 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "How Should Transformers Encode Numeric Values in Electronic Health Records?".

Jane: Here's the gist of the paper. The gist The study systematically compares discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data and real-world clinical prediction tasks to reveal trade-offs between numeric precision, optimisation stability, and architectural flexibility.

Tom: Systematic Evaluation Framework The main contributions of this study include a unified evaluation framework for numeric reasoning in EHR transformers, which introduces a reusable test suite for systematically evaluating numeric value encodings in transformer-based EHR models by combining synthetic arithmetic tasks embedded within real EHR sequences with real-world clinical prediction tasks The framework enables controlled analysis of numeric reasoning capabilities under clinically relevant conditions.

Jane: Comparison of Encoding Strategies The study presents a carefully designed empirical study of discrete, continuous, and hybrid numeric value encodings in transformer-based EHR models, characterizing their trade-offs in numeric precision, optimisation stability, and architectural flexibility using both synthetic arithmetic tasks embedded in real data and real-world clinical prediction.

Tom: The authors show that approaches that explicitly model value-concept interactions provide a strong inductive bias for fine-grained numeric reasoning, while hybrid sequence integrated approaches with binning offer a robust and scalable alternative.

Title and authors: Jane: So, looking at the title "How Should Transformers Encode Numeric Values in Electronic Health Records?", it really frames the whole problem around medical data. It’s not just a general machine learning question; it’s specifically about the unique challenges of EHR data.

Tom: Exactly. And they bring together a few authors here, like Maria Elkjær Montgomery, Christian Igel, Mikkel Odgaard, and Martin Sillesen. It shows this isn't just one person's idea but a team tackling this from different angles.

Lu: From what I see in the context of the paper introduction, they are taking existing methods that people have used for encoding numbers—like using quantile-based binning—and they are testing them against each other in a very structured way.

Meng: So, what's the actual takeaway for us? Does this paper give us a ready-made solution we can just plug in and run without worrying about the underlying architecture?

Lalam: It suggests that instead of just picking one method, you need a framework to systematically compare discrete versus continuous versus hybrid methods using both synthetic tasks and real clinical prediction tasks.

The paper's summary: Tom: They summarize it by saying they introduce a unified evaluation framework for numeric reasoning in EHR transformers. That's a reusable test suite that combines synthetic arithmetic problems with actual EHR sequences and real-world clinical prediction tasks.

Jane: That sounds very practical. So, the core idea is to control the complexity of the math you ask the model by mixing controlled synthetic signals with noisy real data to see how it reacts.

Lu: What's interesting about their summary is that they specifically look at how these encodings perform on precision-sensitive arithmetic tasks, like addition or polynomial evaluation.

Meng: I’m curious about the results on stability. When you’re training something, if your performance degrades smoothly as the task gets harder instead of just crashing when it gets a bit noisy, that makes deployment much more feasible for us.

Lalam: They found that approaches that explicitly model value-concept interactions tend to perform best on those precision-sensitive arithmetic tasks when the architecture allows it.

The paper's improvements: Tom: The paper suggests a few key improvements, and one big one is that joint embedding approaches provide a useful inductive bias for value-sensitive reasoning. This means the AI system gets an extra hint on how values relate to each other, which lets it do complex math like multiplication and polynomial evaluation with higher accuracy.

Jane: That sounds like they are moving beyond just treating numbers as separate tokens and starting to model their relationships directly in the embedding space. That’s a big step for capturing joint significance in clinical data.

Lu: And they also point out that hybrid token-based approaches that apply binning before projection offer a more robust and broadly applicable alternative, especially when architectural constraints are tight. They find this approach is strong under low-data settings because of the binning prior to projection.

Meng: So, if we're constrained by how much compute we have, maybe that hybrid binning route is the sensible way to go for robustness without sacrificing too much performance?

Lalam: They also derived a generalized rule for binning: the optimal number of bins follows a power-law relationship with dataset size, specifically B(N) equals 1 point 14N to the power of zero point two three seven with an R squared of zero point eight three. That gives us a scalable way to preprocess data without needing manual domain knowledge for setting bin counts.

Conclusion: Jane: So, wrapping up the paper "How Should Transformers Encode Numeric Values in Electronic Health Records?", they conclude that while exact arithmetic isn't always needed, models consistently show reliable “good enough” numeric computation across different tasks when precision doesn't demand perfection.

Tom: They also found that for clinical prediction tasks—like breast cancer or lung cancer prediction—the additional methods only produce modest and task-dependent effects compared to the main analysis. It means incorporating laboratory values doesn't automatically lead to huge predictive wins in those specific settings.

Lu: The authors flag a limitation, which is that they couldn't fully isolate the effect of just the encoding strategy because they also varied the surrounding architectural mechanisms used to integrate those numbers.

Meng: That makes sense from an engineering standpoint. It’s hard to say if a performance boost came from the binning or just how we connected that binning into our specific transformer layers.

Lalam: They emphasize that for EHR tasks, robustness and scalability are often more important than exact arithmetic precision, and approaches that integrate numeric information directly into the token sequence without changing the core architecture remain naturally compatible with all transformer models.

Jane: So, to sum up this paper, they give us a controlled evaluation framework to compare discrete versus continuous versus hybrid methods using real clinical data for systematic testing. It’s a solid foundation for future research in this area.

Tom: We get a clear idea that joint encoding strategies like FiLM offer the best trade-off for precision when you can afford the complexity, and hybrid binning is a great fallback if simplicity and scalability are the main drivers.

Lu: It’s interesting how they found that all methods reliably perform approximate numeric computations, which suggests we might be able to get away with "good enough" reasoning for many EHR applications.

Meng: So, the big picture is that we don't need perfect math every time; we just need a system that keeps working and scales well on real traffic.

Lalam: This paper "How Should Transformers Encode Numeric Values in Electronic Health Records?" gives us a clear path forward by providing both the evaluation framework and empirically derived rules for discretization, which is really useful for anyone building these systems.

Maria Elkjár Montgomery, Christian Igel, Mikkel Odgaard, Martin Sillesen, Mads Nielsen

Department of Computer Science, University of Copenhagen · Department of Organ Surgery and Transplantation, Copenhagen University Hospital - Rigshospitalet

cs.LG, cs.AI

Submitted: 2026-07-01

Updated: 2026-07-01

Comments: 16 pages, 15 figures, 3 tables, accepted to ICML 2026, to be published in Proceedings of Machine Learning Research

Journal ref: Proceedings of the 43rd International Conference on Machine Learning (2026)

Code: https://github.com/Montgomeryyyy/BONSAI_values

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The study systematically compares discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data and real-world clinical

Key concepts

Discrete, Continuous, and Hybrid Encodings: These are different ways to represent numbers in a computer model: discrete means using distinct categories (like 1, 2, 3); continuous means using any real number (like 1.5 or -0.7); and hybrid combines both techniques. The study tested how each approach affects the model's ability to perform math and remain stable.
FiLM: FiLM is a method used to explicitly model interactions between values and concepts within the transformer architecture. When used for numeric encoding, it allows the AI to capture complex relationships between different numbers, leading to better accuracy in precise arithmetic tasks like multiplication or polynomial evaluation.
Approximate Numeric Reasoning: This refers to the ability of a model to perform calculations that are 'close enough' rather than perfectly exact. The research shows that most encoding methods can handle this smoothly; performance degrades gradually as math gets harder, suggesting 'good enough' reasoning is often practical for clinical use.
Binning Rule Derivation: This involves finding the optimal number of bins (categories) to use when discretizing continuous data. The study empirically found that the best bin count scales with dataset size according to a power-law relationship, allowing for an automated way to choose how many categories are needed based on how much data is available.

Terminology

Transformer — 2× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 6 episodes
Transformers — 1× in this paper · explained in 2 episodes

Summary

The gist The study systematically compares discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data and real-world clinical prediction tasks to reveal trade-offs between numeric precision, optimisation stability, and architectural flexibility.

Systematic Evaluation Framework

The main contributions of this study include a unified evaluation framework for numeric reasoning in EHR transformers, which introduces a reusable test suite for systematically evaluating numeric value encodings in transformer-based EHR models by combining synthetic arithmetic tasks embedded within real EHR sequences with real-world clinical prediction tasks The framework enables controlled analysis of numeric reasoning capabilities under clinically relevant conditions.

Comparison of Encoding Strategies

The study presents a carefully designed empirical study of discrete, continuous, and hybrid numeric value encodings in transformer-based EHR models, characterizing their trade-offs in numeric precision, optimisation stability, and architectural flexibility using both synthetic arithmetic tasks embedded in real data and real-world clinical prediction. The authors show that approaches that explicitly model value-concept interactions provide a strong inductive bias for fine-grained numeric reasoning, while hybrid sequence integrated approaches with binning offer a robust and scalable alternative.

Characterisation of Precision-Stability Trade-off

The authors show that all evaluated methods can reliably perform approximate numeric computations, with performance degrading smoothly as task complexity increases rather than failing abruptly, suggesting that “good enough” numeric reasoning may be sufficient for EHR applications. They find that clinical gains from incorporating laboratory values are modest and task-dependent in long-term risk prediction, indicating that encoding choices primarily affect robustness and deployability rather than yielding large improvements in predictive performance in this setting.

Performance Across Arithmetic Tasks

For precision-sensitive arithmetic tasks, including addition, multiplication, and polynomial evaluation, FiLM produce the highest scores among the evaluated methods, particularly at lower noise levels and smaller input sizes. The combination method shows intermediate behaviour between FiLM and discretisation-based approaches, suggesting that both continuous value representations and explicit value-concept interactions contribute to performance on precision-sensitive tasks.

Implications for Clinical Prediction

Clinical results for all methods are shown in Figure 15, where across breast cancer, lung cancer, and stroke prediction, the additional methods produce modest and task-dependent effects similar to those observed in the primary analysis. The study concludes that when fine-grained numeric precision is required, joint encoding methods such as FiLM provide a strong overall trade-off between precision and robustness. In settings where architectural simplicity or scalability are primary concerns, hybrid token-based approaches such as combined binning offer a strong overall compromise.

General Binning Rule Derivation

To derive a generalised binning rule, the optimal bin count is found to follow a power-law relationship between dataset size and optimal bin count, yielding B(N) = 1.14N 0.237 with R2 = 0.83. This empirical approximation suggests that the optimal number of bins follows a simple empirically derived power-law in dataset size.

Robustness and Scalability

The authors suggest that for EHR-style tasks, approximate numeric reasoning may be sufficient, and that robustness and scalability are often more important than exact arithmetic precision. Approaches that integrate numeric information directly into the token sequence without modifying the underlying architecture remain naturally compatible with encoder-only, decoder-only, and multimodal transformer models. The study introduces a simple discretisation heuristic whereby the empirically optimal number of bins approximately follows a power-law relationship with dataset size.

Controlled Evaluation Framework

An additional contribution of this work is the introduction of a controlled evaluation framework for studying numeric value encodings approaches in transformer-based EHR models, enabling systematic comparison across settings requiring different forms of numeric reasoning, robustness, and scalability. This evaluation framework is made publicly available to support future research on numeric encodings in clinical transformers.

Limitations

The study has several limitations, including that the evaluated methods differ not only in how numeric values are encoded, but also in the surrounding architectural mechanisms used to integrate them, making it difficult to fully isolate the effect of the encoding strategy itself. Furthermore, while synthetic arithmetic tasks enable controlled evaluation of numeric reasoning behaviour, they do not fully capture the complexity of real-world clinical prediction. This suggests that multiple encoding strategies can provide sufficiently robust numeric reasoning for many EHR applications.

Acknowledgements and Impact

The work was supported by the Danish National Research Foundation, grant no. P1, and the Novo Nordisk Foundation, grant no. NNF23OC0083562. Machine learning applied to electronic health records has the potential to make healthcare data more useful to clinicians, patients, and administrators. Any downstream deployment would require careful validation across the intended patient population, as models trained on EHR data may inherit biases present in healthcare systems.

Improvements for AI systems

Continuous performance gains on precision-sensitive arithmetic tasks are achieved by using FiLM to explicitly model value-concept interactions, as joint embedding approaches provide a useful inductive bias for value-sensitive reasoning. This allows the AI system to perform complex arithmetic operations like multiplication and polynomial evaluation with higher accuracy.
Robustness in low-data regimes is enhanced by employing hybrid token-based approaches with binning prior to projection, which offers a robust and scalable alternative when architectural constraints permit. This enables the AI system to maintain reliable numeric computation even when encountering sparse or heterogeneous laboratory values.
The system can be designed to automatically determine the optimal number of bins for discretization by fitting an empirically derived power-law heuristic, as the optimal bin count follows a simple empirically derived power-law in dataset size. This provides a scalable method for preprocessing EHR numeric data without relying on manual domain knowledge.
The AI system will exhibit reliable approximate numeric reasoning, as all evaluated methods can reliably perform approximate numeric computations, with performance degrading smoothly rather than failing abruptly as task complexity increases. This ensures the system remains operational and provides useful insights even when exact arithmetic precision is not achievable in high-complexity scenarios.
In clinical prediction tasks, the AI system will benefit from incorporating laboratory values, though clinical gains from incorporating laboratory values are task-dependent. The encoding choice primarily affects robustness and deployability rather than yielding large improvements in predictive performance in this setting.