Large Language Models Naively Recover Ethnicity from Individual Records

arXiv:2601.21132 · cs.CL · Submitted 2026-01-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Large Language Models Naively Recover Ethnicity from Individual Records".

Jane: The gist: Large language models can infer ethnicity from names with accuracy exceeding that of Bayesian Improved Surname Geocoding (BISG) without additional training data,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're starting with "Large Language Models Naively Recover Ethnicity from Individual Records". This paper is looking at how language models can figure out ethnicity just from a name and a location, without needing any extra training data.

Jane: Right, so the big claim here is that these large language models can guess someone’s ethnic background by only looking at their name and where they live, which is useful for contexts outside the United States.

Tom: It's not just about guessing; it's about beating established methods like BISG on some US voter files. The authors say with stratification from Florida and North Carolina voter files, the LLM-based classification reaches up to eighty-four point seven percent accuracy, outperforming BISG which gets sixty-eight point two percent >

Jane: That accuracy number is significant because it shows that this method works better than what we currently use for this kind of demographic inference, and they tested six different models including Gemini three Flash and GPT-4o >

Tom: I'm seeing how the accuracy changes depending on what you feed the model, like whether you give it just a name versus a name along with a location. The paper shows that removing first names reduces GPT-4o accuracy by zero point six percentage points in Florida >

Jane: And they also found something interesting about income bias, suggesting these LLM classifications reduce the bias inherent in BISG where minorities living in wealthier neighborhoods get misclassified as White >

Tom: It’s about how these models handle those complex relationships between names and location data, moving away from just relying on old geographic racial priors >

Jane: This opens up a whole new way for us to look at demographic information that isn't tied down by traditional databases, which is really valuable for researchers trying to study representation >

Tom: And for anyone listening who doesn't read the math, it means we can potentially get more accurate population distributions from records we already have, even in places outside the United States >

Jane: Exactly. They validated this approach across different contexts too, using Lebanese voter registration records for religious sect and Indian data for caste labels >

Tom: They also did aggregate validation by comparing their inferred ethnic distributions to census data across six other countries like India, Uganda, Nepal, Armenia, Chile, and Costa Rica >

Jane: And they noted that in Lebanon with religious sect records they achieved sixty-four point three percent accuracy for groups with distinctive naming conventions like Armenian Orthodox >

Tom: The paper also tested how extended reasoning improves things; it said that enabling this reasoning can improve accuracy by one to three percentage points, though the effects vary depending on the context >

Jane: That variation is important because it suggests that you can't just assume extended reasoning will help every single time you use these models >

Tom: And they also found that metadata like party registration reaches an eighty-six point seven percent accuracy, which shows how much extra information helps the model sort things out >

Jane: So what does this mean for the next step in understanding this work? It suggests that we can start making these kinds of inferences without needing massive labeled datasets right away >

Tom: The paper also points out that for large-scale applications, you can fine-tune smaller transformer models on these LLM labels, and those fine-tuned models could approach the teacher accuracy on US data while allowing for local deployment at no cost >

Jane: That's a practical point—it means researchers don't have to rely solely on massive proprietary models to get these kinds of classifications if they want something deployable >

Tom: So we've seen how this paper, "Large Language Models Naively Recover Ethnicity from Individual Records", offers a path to using existing records for demographic inference with accuracy exceeding BISG on US data and across other regions >

Jane: That means the core claim is that the naming patterns themselves have enough signal for an AI to do the work, which is what was missing in older systems that needed those specific Census or voter file links >

Tom: The authors tested models like Gemini three Flash and GPT-4o and found they can get up to eighty-four point seven percent accuracy compared to the sixty-eight point two percent BISG hit on balanced samples >

Jane: And they proved it’s not just about one place; they validate this across Lebanon using religious sect records and India with caste labels >

Jane: And they also noted that in Lebanon with religious sect records they achieved sixty-four point three percent accuracy for groups with distinctive naming conventions like Armenian Orthodox >

Conclusion: Tom: So we're wrapping up this look at "Large Language Models Naively Recover Ethnicity from Individual Records."

Jane: Right, so what this paper really boils down to is that these large language models can guess someone’s ethnicity just by looking at their name and where they live without needing any extra training data.

Tom: It takes existing names and geography and feeds them into a model to get a classification that actually beats traditional methods like BISG on US voter files.

Tom: The authors tested models like Gemini three Flash and GPT-4o, and they found that even with just names and location, you can get up to eighty-four point seven percent accuracy compared to the sixty-eight point two percent BISG hit on those balanced samples.

Jane: And they proved it’s not just about one place; they validate this across Lebanon using religious sect records and India with caste labels.

Tom: It extends to six other countries too, comparing their inferred distributions to census data using original full-count voter rolls.

Jane: And they also noted that this method doesn't just reproduce old biases; for instance, the Gemini results showed no income bias gradient when matched to neighborhood wealth in North Carolina.

Tom: So what does this mean for the bigger picture? It means we can infer ethnicity from names and location even outside the United States with high accuracy without needing massive labeled datasets for training.

Jane: That opens up a path where you can get demographic information from records you already have, which is a huge shift in how we think about population data.

Tom: And while they showed that fine-tuning smaller models on these labels can work and even allow for local deployment, there are still questions about exactly how reliable these inferences are when dealing with very small groups or highly specific naming conventions.

Jane: That’s a big question—how good is the guess when the data is sparse? We’ll get into that next.

Harvard University

cs.CL

Submitted: 2026-01-29

Updated: 2026-10-15

Comments: This paper has been superseded by arXiv:2610.19048 (merged from arXiv:2601.21132 and arXiv:2601.21138)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: Large language models can infer ethnicity from names with accuracy exceeding that of Bayesian Improved Surname Geocoding (BISG) without additional training data, enabling inference outside

Key concepts

Large Language Models (LLMs): These are sophisticated AI programs trained on massive amounts of text to understand context and relationships. In this research, they are prompted with a name and location to predict an ethnicity category based purely on the linguistic patterns within the name itself.
Bayesian Improved Surname Geocoding (BISG): This is a traditional statistical method used to predict ethnicity by combining surname information with geographic data. The LLM approach shows it can achieve higher accuracy than BISG because it can use more complex contextual clues from names, even when traditional methods struggle.
Contextual Classification: This means classifying a person into an ethnic group based on the specific setting or category being considered. The LLM method excels here by moving beyond simple race/ethnicity to classify groups like religious sects in Lebanon or caste in India, which BISG cannot easily handle.

Terminology

Accuracy — 5× in this paper · explained in 1 episode
Bayesian Improved Surname Geocoding — 1× in this paper
Transformer — 1× in this paper · explained in 2 episodes
Transformer Model — 1× in this paper · explained in 2 episodes
Transformer Models — 1× in this paper · explained in 2 episodes

Summary

The gist: Large language models can infer ethnicity from names with accuracy exceeding that of Bayesian Improved Surname Geocoding (BISG) without additional training data, enabling inference outside the United States and to contextually appropriate classification categories.

How it works

The approach involves prompting a large language model to classify names into specified ethnic categories based on the name and location provided in the prompt, either alone or with additional meta information like geographic location<ref:2601.21132[cs.CL] 3>. A typical prompt takes the form “Classify the race/ethnicity of this person based on their name and location. Name: [name]. Location: [geography]. Return only one of: [categories].” The geography varies by context, defined at the county level for American voter files and at the region or province level elsewhere<ref:2601.21132[cs.CL] 4>.

Model Comparison and Performance

The research tested six models including Gemini 3 Flash, GPT-4o, DeepSeek v3.2, GLM-4.7, and GLM-4.7-Flash to assess performance variation across model families and sizes<ref:2601.21132[cs.CL] 5>. In Florida and North Carolina using stratified samples of 10,000 records (2,500 per racial category), Gemini achieved 83.8% accuracy with full names and geography, compared to 68.2% for BISG with surname and county<ref:2601.21132[cs.CL] 6>. The performance of LLMs varied significantly based on input components; removing first names reduced GPT-4o accuracy by 0.6pp in Florida, while removing both first names and geography reduced accuracy by 8.7pp<ref:2601.21132[cs.CL] 6>.

Validation Across Diverse Contexts

The method was validated across several contexts, including individual-level comparisons with Florida and North Carolina voter files, Lebanon where religious sect is recorded in voter registration, and India using Members of Parliament from reserved constituencies and land records with caste labels<ref:2601.21132[cs.CL] 7>. Aggregate validation compared inferred ethnic distributions to census data across six additional countries: India, Uganda, Nepal, Armenia, Chile, and Costa Rica using original full count voter rolls<ref:2601.21132[cs.CL] 7>. In Lebanon, the LLMs achieved high accuracy for groups with distinctive naming conventions such as Armenian Orthodox (up to 97.4% for Gemini) and Shia<ref:2601.21132[cs.CL] 8>.

Addressing Limitations and Scaling

A key advantage of LLM-based classification over BISG is the ability to incorporate additional information, such as first and middle names, which carry substantial ethnic signal<ref:2601.21132[cs.CL] 12>. The approach circumvents barriers like lack of Census surname data or legal prohibitions on collecting racial data in certain countries<ref:2601.21132[cs.CL] 4>. For large-scale applications, researchers can use LLM classification to create training sets and then fine-tune lightweight models on these labels using knowledge distillation experiments, which showed that fine-tuned transformers could approach teacher accuracy on US data while enabling local deployment<ref:2601.21132[cs.CL] 9>.

Income Bias Analysis

The LLM approach also reduces the income bias inherent in BISG, where minorities in wealthier neighborhoods are systematically misclassified as White<ref:2601.21132[cs.CL] 7>. Replicating prior analysis on North Carolina voter records matched to Census Bureau median household income by ZIP Code Tabulation Area (ZCTA), the results showed that while BISG misclassification rises sharply with neighborhood income for Black voters, Gemini showed no such gradient<ref:2601.21132[cs.CL] 17>. This differential bias likely arises because BISG relies on geographic racial composition as a prior, whereas LLMs incorporate geographic metadata in a less strictly defined manner<ref:2601.21132[cs.CL] 7>.

Reasoning and Robustness

The effect of enabling extended reasoning on classification accuracy was mixed; for instance, Gemini 3 Flash gained 1.0 percentage point in Florida, while Qwen 3 8B gained a larger gain of 3.4 percentage points in North Carolina<ref:2601.21132[cs.CL] 14>. However, this inconsistent result across states and models suggests that extended reasoning may not reliably improve classification and substantially increases inference time<ref:2601.21132[cs.CL] 14>. Overall, the method provides a viable alternative to BISG for ethnicity inference from names, achieving comparable or superior accuracy without requiring additional labeled data<ref:2601.21132[cs.CL] 8>.

Conclusion and Recommendations

The LLM approach extends naturally to classification schemes that BISG cannot address, including religious sect, caste, and ethnicity, and to countries lacking detailed surname-geographic probability distributions<ref:2601.21132[cs.CL] 8>. Researchers should validate before deployment by handcoding a small sample or comparing inferred distributions to available benchmarks<ref:2601.21132[cs.CL] 8>. The results also reveal that model selection involves a trade-off between accuracy and reproducibility, suggesting that open-source or researcher fine-tuned models are likely preferable for applications where results must be verifiable and replicable<ref:2601.21132[cs.CL] 8>.

References

Argyle, Lisa P., and Michael Barber. “Misclassification and Bias in Predictions of Individual Ethnicity from Administrative Records.” American Political Science Review 118, no. 2 (2024): 1058–1066<ref:2601.21132[cs.CL] 19>.

Chintalapati, Rajashekar, Suriyan Laohaprapanon, and Gaurav Sood. “Predicting Race and Ethnicity from the Sequence of Characters in a Name.” arXiv preprint arXiv:1805.02109 (2018)<ref:2601.21132[cs.CL] 4>.

Greengard, Philip, and Andrew Gelman. “A Calibrated BISG for Inferring Race from Surname and Geolocation.” Journal of the Royal Statistical Society Series A: Statistics in Society (2025)<ref:2601.21132[cs.CL] 3>.

Imai, Kosuke, and Kabir Khanna. “Improving Ecological Inference by Predicting Individual Ethnicity from Voter Registration Records.” Political Analysis 24, no. 2 (2016): 263–272<ref:2601.21132[cs.CL] 8>.

Imai, Kosuke, Santiago Olivella, and Evan T. R. Rosenman. “Addressing Census Data Problems in Race Imputation via Fully Bayesian Improved Surname Geocoding and Name Supplements.” Science Advances 8, no. 49 (2022): eadc9824<ref:2601.21132[cs.CL] 3>.

Khanna, Kabir, Brandon Bertelsen, Santiago Olivella, Evan Rosenman, Alexander Rossell Hayes, and Kosuke Imai. “wru: Who are You? Bayesian Prediction of Racial Category Using Surname, First Name, Middle Name, and Geolocation.” R package version 3.0.3 (2024)<ref:2601.21132[cs.CL] 3>.

Lee, Diana Da In, and Yamil R. Velez. “Measuring Descriptive Representation at Scale: Methods for Predicting the Race and Ethnicity of Public Officials.” British Journal of Political Science 55, e110 (2025)<ref:2601.21132[cs.CL] 7>.

Spirling, Arthur. “Why Open-Source Generative AI Models Are an Ethical Way Forward for Science.” Nature 616, no. 7957 (2023): 413<ref:2601.21132[cs.CL] 8>.

Xie, Fangzhou. “Rethnicity: An R Package for Predicting Ethnicity from Names.” SoftwareX 17 (2022): 100965<ref:2601.21132[cs.CL] 4>.

Ye, Junting, Shuchu Han, Yifan Hu, Baris Coskun, Meizhu Liu, Hong Qin, and Steven Skiena. “Nationality Classification Using Name Embeddings.” In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 1897–1906 (2017)<ref:2601.21132[cs.

Improvements for AI systems

Bold names can be inferred without labeled training data, enabling ethnicity inference in countries where no Census surname data exists and for categorization schemes researchers define to match their research questions.
The system can incorporate additional metadata such as first and middle names that carry substantial ethnic signal, geographic information at multiple hierarchical levels, and additional metadata such as sex, age, or party registration.
The improved AI system can reduce the income bias inherent in traditional methods, specifically addressing how BISG causes minorities in wealthier neighborhoods to be misclassified as White by showing that LLM classification also reduces the income bias inherent in BISG.
The system can perform validation across diverse, non-US contexts by comparing inferred distributions to census data using original fullcount voter rolls from countries like India, Uganda, and Nepal.
Small transformer models fine-tuned on LLM labels can be used for local deployment at no cost, offering a scalable solution for population-level inference that exceeds BISG accuracy while being runnable on consumer hardware.
Extended reasoning modes in large language models can improve accuracy by 1–3 percentage points, allowing the system to perform intermediate reasoning steps before generating a final answer on complex classification tasks.

Abstract

I demonstrate that large language models can infer ethnicity from names with accuracy exceeding that of Bayesian Improved Surname Geocoding (BISG) without additional training data, enabling inference outside the United States and to contextually appropriate classification categories. Using stratified samples from Florida and North Carolina voter files with self-reported race, LLM-based classification achieves up to 84.7% accuracy, outperforming BISG (68.2%) on balanced samples. I test six models including Gemini 3 Flash, GPT-4o, and open-source alternatives such as DeepSeek v3.2 and GLM-4.7. Enabling extended reasoning can improve accuracy by 1-3 percentage points, though effects vary across contexts; including metadata such as party registration reaches 86.7%. LLM classification also reduces the income bias inherent in BISG, where minorities in wealthier neighborhoods are systematically misclassified as White. I further validate using Lebanese voter registration with religious sect (64.3% accuracy), Indian MPs from reserved constituencies (99.2%), and Indian land records with caste classification (74.0%). Aggregate validation across India, Uganda, Nepal, Armenia, Chile, and Costa Rica using original full-count voter rolls demonstrates that the method recovers known population distributions where naming conventions are distinctive. For large-scale applications, small transformer models fine-tuned on LLM labels exceed BISG accuracy while enabling local deployment at no cost.

Sources

Predicting Race and Ethnicity From the Sequence of Characters in a Name