Using Embedding Models to Improve Probabilistic Race Prediction

arXiv:2604.22555 · cs.CL · Submitted 2026-04-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Using Embedding Models to Improve Probabilistic Race Prediction".

Jane: Here's the gist of the paper. The gist: The proposed embedding-powered BISG (eBISG) methodology uses pre-trained text embeddings to represent names as dense vectors and trains neural networks to estimate race probabilities for names not covered in the Census, significantly improving predictive performance for individuals with uncommon surnames.

Tom: Problem Addressed Estimating racial disparity requires individual-level race data, which are often unavailable due to the sensitivity of collecting such information.

Jane: The widely adopted Bayesian Improved Surname Geocoding (BISG) critically relies on Census surname data, omitting approximately 10% of the US population with uncommon surnames.

Tom: This omission causes predictive performance to degrade substantially for individuals with such omitted, uncommon surnames because standard BISG implementation relies on an uninformative generic prior in these cases.

Title and authors: Tom: So we’re diving into "Using Embedding Models to Improve Probabilistic Race Prediction" today. We're looking at how they took a problem where you don't have individual race data and tried to fix it using text embeddings.

Jane: Right, so the core idea is they are using these pre-trained text embeddings to represent names as dense vectors, which lets them train neural networks to estimate race probabilities for names that aren't actually in the Census.

Lu: It shows that you can take an existing method like BISG and augment it by replacing that generic prior with specific, name-based race probabilities derived from these embeddings. It moves the system from guessing to learning from linguistic structure.

Meng: From my side, I’m interested in how they handle the scale of those embeddings; if we have millions of names, we need an embedding generation process that doesn't choke our deployment pipeline.

Tom: Exactly, and they show this isn't just theoretical work; they compare five different approaches to see which one performs best when you start adding more name information to the model.

Jane: And what’s really interesting is how this approach reduces those systematic, income-correlated biases that researchers like Argyle and Barber found in older disparity estimates.

Lu: They show that by using full-name embeddings, which treat the first, middle, and last name as one single representation, they capture interactions between name components that standard models miss.

Meng: That captures the interaction aspect is important for real-world fairness because it moves beyond just looking at one part of a name.

Tom: It means if you use this full-name approach, the error tied to someone’s wealth level shrinks substantially in those disparity estimates.

Jane: So if you adopt this method, the error tied to income level doesn't just disappear; it gets much smaller across different income groups.

Lu: They found that the full-name embedding model achieved the highest precision at every recall level for every race group they tested by modeling the full name as a single representation that captures interactions between name components.

Meng: But they did have to be careful about the training data; it was based on voter files from four Southern states, and they flagged a distributional shift as a major limitation for broader use.

Tom: So we’ve seen the mechanism: embeddings map names to vectors, those vectors train a neural network to estimate race probabilities for missing Census data.

Jane: It changes things because it shows that the structure of the name itself has predictive power that standard administrative data simply ignores when it comes to race.

Lu: They are essentially replacing a statistical assumption with learned linguistic representations, which is a big move in representation learning theory.

Meng: For deployment, I’d be looking at the computational cost of generating those embeddings at scale, because we don't want to run heavy models for every single name query.

Tom: That’s fair; the engineering reality is that high-dimensional vectors can get expensive when you need them instantly.

Jane: But the paper is really pushing toward a model where you can use this approach to augment existing tools like fBISG without having to completely rewrite everything downstream.

Lu: They explicitly designed it to integrate with those existing frameworks, which shows they thought about the practical ecosystem of these tools and how they would actually be used.

Meng: I’m still skeptical about the generalizability because that distributional shift they mentioned is a real hurdle for production systems.

Tom: So in short, "Using Embedding Models to Improve Probabilistic Race Prediction" shows how richer name representations can significantly improve accuracy and fairness when Census data is missing for uncommon surnames.

Jane: It’s a solid piece of work that proves representation learning has tangible benefits when applied to demographic prediction problems.

Lu: It opens up the idea that we can leverage external, large-scale text models to inject contextual information into traditional statistical inference tasks.

Meng: We'll keep an eye on how they handle the scaling of those embeddings, because that’s where I think the next set of papers will have to focus if this is going to move beyond the lab.

The paper's summary: Tom: So we're looking at how they actually summarize the paper "Using Embedding Models to Improve Probabilistic Race Prediction" now and what that means for us overall in terms of predictive tools.

Jane: Right, so essentially, they're showing us how taking those name embeddings and feeding them into a neural network lets you get individual race probabilities even for names that aren't on the Census. It’s about turning a missing piece of data into something that helps with prediction.

Lu: What they really emphasize is the comparison of five different ways to do this, showing how adding more name detail improves results and which method ends up being best for capturing those name component interactions.

Meng: And I see them specifically focusing on how the full-name embedding version gives substantially larger gains across all race categories compared to just using just surnames. That's a big win in terms of accuracy improvement.

Tom: Exactly, and they’re pointing out that this full-name approach actually reduces those income-related biases we talked about because you get rid of that generic prior assumption when the name is incomplete.

Jane: So if you use this method, the error tied to someone’s wealth level gets much smaller in your disparity estimates. It makes those disparities look more accurate across different income groups.

Lu: They found that the full-name embedding model actually achieved the highest precision at every recall level for every race group they tested because it treats the whole name as one representation capturing how the parts interact.

Meng: But I have to bring up that training data point again, because they had to be careful—it was voter files from four Southern states, and they flagged a distributional shift as a major issue when you try to use it elsewhere.

Tom: So we’ve seen the mechanism: embeddings map names to vectors, those vectors train a neural network to estimate race probabilities for missing Census data. It’s turning name structure into predictive power for these systems.

Jane: It changes things because it shows that the structure of the name itself has predictive power that standard administrative data simply ignores when it comes to race. It’s adding external knowledge where the original source falls short.

Lu: They are really replacing a statistical assumption with learned linguistic representations, which is a big move for representation learning theory in this field.

Meng: For us, the big question remains the scaling of those embeddings; we need to figure out how to generate them efficiently so we can actually use this in systems processing millions of names without needing massive computational resources for every query.

Tom: That’s fair; the engineering reality is that high-dimensional vectors can get expensive when you need them instantly. So the paper suggests a path forward for making these predictive tools more robust against data sparsity and inherent biases built into the Census. It’s about building smarter predictors from what we already have, plus some external context.

Jane: It’s a solid piece of work because it proves representation learning has tangible benefits when applied to demographic prediction problems that are otherwise stuck with incomplete administrative data. We've seen the mechanism and what it changes for the system.

Lu: It opens up the idea that we can leverage external, large-scale text models to inject contextual information into traditional statistical inference tasks. That’s a big step toward using general language knowledge in specialized areas like demographic prediction problems.

The paper's improvements: Tom: So we’re looking at the specific ways "Using Embedding Models to Improve Probabilistic Race Prediction" suggests we improve things compared to just using standard methods for finding race probabilities in uncommon surnames.

Jane: Right, and the core idea is that instead of relying on a weak generic guess when a name isn't in the Census, they use those vectors from the embeddings to train something that actually predicts race probabilities for those missing names. It’s about turning a missing piece of data into something that helps with prediction.

Lu: What’s really interesting is how they compare five different ways to do this—from just standard BISG up to using full-name embeddings trained on voter file data. They show you the evolution of adding more name information, and which one wins.

Tom: Exactly, and they also showed that this full-name version reduced those systematic income-correlated biases identified earlier because the uninformative prior is gone. The bias tied to wealth level shrinks substantially when you use this approach instead of the default setting.

Jane: That means if you use this method, the error tied to someone’s wealth level doesn't just disappear; it gets much smaller in your disparity estimates. It makes those disparities look more accurate across different income groups.

Lu: They found that the full-name embedding model achieved the highest precision at every recall level for every race group they tested because it treats the whole name as one representation capturing how the parts interact.

Meng: But they did have to be careful about that training data again—it was voter files from four Southern states, and they flagged a distributional shift as a major limitation for broader use. That’s a real hurdle for production systems that need to work reliably everywhere.

Jane: But the paper is really pushing toward a model where you can use this approach to augment existing tools like fBISG without having to completely rewrite everything downstream. It’s about adding intelligence on top of what's already working.

Lu: They explicitly designed it to integrate with those existing frameworks, which shows they thought about the practical ecosystem of those tools and how they would actually be used.

Meng: For deployment, I’m looking at the computational cost of generating those embeddings at scale because we don't want to run heavy models for every single name query. That’s a key question for real-world deployment.

Tom: So in short, this paper shows how richer name representations can significantly improve accuracy and fairness when Census data is missing for uncommon surnames. It’s a strong demonstration of how representation learning can help fix specific gaps in administrative data.

Jane: It’s a solid piece of work that proves representation learning has tangible benefits when applied to demographic prediction problems that are otherwise stuck with incomplete administrative data. We've seen the mechanism and what it changes for the system.

Lu: They open up the idea that we can leverage external, large-scale text models to inject contextual information into traditional statistical inference tasks. That’s a big step toward using general language knowledge in specialized areas like demographic prediction problems.

Conclusion: Tom: So we’ve covered how "Using Embedding Models to Improve Probabilistic Race Prediction" uses pre-trained text embeddings to fix a problem in standard surname geocoding for people with uncommon names.

Jane: Right, and the core idea is that instead of relying on a weak generic guess when a name isn't in the Census, they use those vectors from the embeddings to train something that actually predicts race probabilities. It’s about turning a missing piece of data into something that helps with prediction.

Meng: I saw them comparing the full-name embedding approach against others, and they found that it produced substantially larger gains across all racial categories. That’s a big win in terms of accuracy improvement.

Meng: But they did have to be careful about that training data—it was voter files from four Southern states, and they flagged a distributional shift as a major limitation for broader use. That’s a real hurdle for production systems that need to work reliably everywhere.

Jane: It changes things because it shows that the structure of the name itself has predictive power that standard administrative data simply ignores when it comes to race.

Lu: They are essentially replacing a statistical assumption with learned linguistic representations, which is a big move in representation learning theory.

Meng: For deployment, I’d be looking at the computational cost of generating those embeddings at scale because we don't want to run heavy models for every single name query. That’s a key question for real-world deployment.

Tom: That’s fair; the engineering reality is that high-dimensional vectors can get expensive when you need them instantly.

Lu: They explicitly designed it to integrate with those existing frameworks, which shows they thought about the practical ecosystem of those tools and how they would actually be used.

Meng: I’m still skeptical about the generalizability because that distributional shift they mentioned is a real hurdle for production systems.

Jane: It’s a solid piece of work that proves representation learning has tangible benefits when applied to demographic prediction problems.

Lu: It opens up the idea that we can leverage external, large-scale text models to inject contextual information into traditional statistical inference tasks.

Meng: We'll keep an eye on how they handle the scaling of those embeddings, because that’s where I think the next set of papers will have to focus if this is going to move beyond the lab.

Noah Dasanaike, Kosuke Imai

Harvard University

cs.CL

Submitted: 2026-04-24

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: The gist: The proposed embedding-powered BISG (eBISG) methodology uses pre-trained text embeddings to represent names as dense vectors and trains neural networks to estimate race probabilities for

Key concepts

eBISG: This methodology replaces the standard Bayesian Improved Surname Geocoding (BISG) approach. It uses pre-trained text embeddings to represent names as vectors, which then inform neural networks to predict race probabilities for names not in the Census, overcoming limitations of older methods.
Text Embeddings: These are dense numerical representations of text, generated by models like E5-Large. They map character strings (like names) into high-dimensional spaces where names with similar characteristics are positioned close together, capturing semantic relationships between name components.
Full-Name Embedding: This technique combines the first, middle, and last name into a single string before embedding it. This allows the model to capture interactions between different parts of a full name that standard methods ignore, leading to better predictions for individuals with uncommon surnames.

Terminology

Bayesian Improved Surname Geocoding — 1× in this paper
Embeddings — 1× in this paper · explained in 2 episodes

Summary

The gist: The proposed embedding-powered BISG (eBISG) methodology uses pre-trained text embeddings to represent names as dense vectors and trains neural networks to estimate race probabilities for names not covered in the Census, significantly improving predictive performance for individuals with uncommon surnames.

Problem Addressed

Estimating racial disparity requires individual-level race data, which are often unavailable due to the sensitivity of collecting such information. The widely adopted Bayesian Improved Surname Geocoding (BISG) critically relies on Census surname data, omitting approximately 10% of the US population with uncommon surnames. This omission causes predictive performance to degrade substantially for individuals with such omitted, uncommon surnames because standard BISG implementation relies on an uninformative generic prior in these cases.

Proposed Methodology: eBISG

The eBISG methodology replaces the uninformative generic prior for unmatched surnames with informative, name-specific race probabilities derived from embeddings. The approach uses open-source pre-trained text embedding models to represent names as dense vectors in a high-dimensional space.

The methodology evaluates five approaches:

Standard BISG using only surnames.
BIFSG incorporating first name probabilities.
Surname embedding for unlisted names.
Surname and first name embedding combining both.
Full-name embedding trained on voter file data from Southern states that captures interactions between name components.

Model Training and Representation

The paper utilizes pre-trained multilingual text embedding models like E5-Large to map character strings into high-dimensional space. The neural networks are trained on Census name data to estimate race probabilities for names not covered in the Census.

The full-name embedding version is trained on individual voter records with self-reported race from state voter registration files. This model concatenates first, middle, and last name into a single string before embedding to capture interactions between name components.

Empirical Evaluation

The eBISG methods are evaluated using random samples of one million voters from North Carolina and Florida where self-reported race provides ground truth. The results are qualitatively similar across the two states, with the ranking of methods preserved in every case.

Key findings from the evaluation include:

**)&The approach based on surname embeddings yields consistent improvements over standard BISG for voters with unmatched names, while the method, which leverages full-name embeddings, produces substantially larger gains across all racial categories in both states. The full-name approach also reduces the systematic, income-correlated bias identified by Argyle and Barber (2024): where standard BISG overpredicts white share in wealthy Census tracts and underpredicts minority share across the income distribution, the full-name embeddings approach reduces these biases toward zero. The full-name embedding model achieves the highest precision at every recall level for every race, by modeling the full name as a single representation that captures interactions between name components. The full-name embedding model provides the best calibration across all racial groups. The systematic, income-correlated misclassification that concerns Argyle and Barber (2024) appears to be at least in a large part a consequence of the uninformative generic prior rather than a fundamental limitation of the BISG framework: once the prior is replaced with an informative name-based prediction, the income gradient substantially attenuates. The full-name embedding model achieves the best calibration for Asian voters while BIFSG is best calibrated for Black voters. The full-name model is trained on voter files from four Southern states whose racial composition differs from the two evaluation states. The distributional shift limits generalization. The methods we propose are designed to integrate with the existing ecosystem of tools built on BISG posteriors, including fBISG (Imai et al., 2022) and BIRDiE (McCartan et al., 2025), without requiring modifications to those downstream methods. The proposed eBISG approach has limitations that merit emphasis.

Improvements for AI systems

This improved system can estimate individual-level race probabilities for individuals with surnames not present in Census data by replacing the uninformative generic prior with informative, name-specific race probabilities derived from embeddings.
The system can capture interactions between name components that the conditional independence assumption precludes by using a full-name embedding approach, which is trained on voter files to represent the first, middle, and last name into a single representation.
The system can reduce systematic, income-correlated bias identified by Argyle and Barber (2024) in disparity estimates because the full-name embeddings show that "the systematic, income-correlated misclassification that concerns Argyle and Barber (2024) appears to be at least in a large part a consequence of the uninformative generic prior rather than a fundamental limitation of the BISG framework."
The system can achieve superior predictive performance for Hispanic and Asian voters whose surnames are absent from the Census, as demonstrated by the finding that the full-name embedding yielding the largest gains, particularly for Hispanic and Asian voters whose surnames are absent from the Census list.

Abstract

Estimating racial disparity requires individual-level race data, which are often unavailable due to the sensitivity of collecting such information. To address this problem, many researchers utilize Bayesian Improved Surname Geocoding (BISG), which have critically relied on Census surname data. Unfortunately, these data capture race-surname relationships only for common surnames, omitting approximately 10% of the US population. We show that predictive performance degrades substantially for individuals with such omitted, uncommon surnames because standard BISG implementation relies on a uninformative generic prior in these cases. To address this limitation, we propose embedding-powered BISG (eBISG), which uses pre-trained text embeddings to represent names as dense vectors and trains neural networks on 2020 Census surname and first-name data to estimate race probabilities for names not covered in the Census. We compare five approaches: standard BISG using only surnames, BIFSG incorporating first name probabilities, surname embedding for unlisted names, surname and first name embedding combining both, and a full-name embedding trained on voter file data from Southern states that captures interactions between name components. We show that each successive eBISG approach improves race prediction, with the full-name embedding yielding the largest gains, particularly for Hispanic and Asian voters whose surnames are absent from the Census list.