Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples

arXiv:2509.25063 · cs.CY, cs.CL · Submitted 2025-09-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Learning from Convenience Samples".

Jane: Here's the gist of the paper. The gist The fine-tuning of Large Language Models (LLMs) on available data allows them to impute self-reported vote choice under both random and systematic nonresponse, suggesting they offer a promising strategy for researchers working with non-probability samples or systematic missingness in surveys.

Tom: How it works The study investigates using LLMs to simulate respondents via persona-based prompts and fine-tuning them on available data to impute self-reported vote choice under both random and systematic nonresponse, using the German Longitudinal Election Study The researchers compare zero-shot prompting and supervised fine-tuning against tabular classifiers (e.g., CatBoost) and test how different convenience samples (e.g., students) used for fine-tuning affect generalization The methodology involves several key steps for the experiments 1.

Jane: Survey Data Processing involved using the German Longitudinal Election Study (GLES) Cross-Section Cumulation 2009-2017 dataset for our experiments (GLES 2020), which is openly available for academic research 2.

Tom: Item Selection Survey items were selected based on factors commonly associated with vote choice in German elections (Schmitt-Beck et al.

Paper summary: Tom: So we're talking about this paper called "Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples". This paper tackles a big problem in survey research: the rising costs of probability samples and all that missing data, like non-response or people dropping out.

Jane: Right, so the core idea here is using large language models, or LLMs, to simulate responses when you don't have perfect data. The authors are looking at a more real-world scenario where you have partial survey answers and trying to figure out vote choice in Germany using the German Longitudinal Election Study.

Lu: They set out to compare two main approaches: zero-shot prompting and supervised fine-tuning against established methods like CatBoost classifiers. The paper claims that they can recover reliable population-level inferences even when dealing with socially or demographically biased data and missingness.

Meng: Okay, so the claim is that fine-tuned LLMs can do better than just using a pre-trained model with a simple prompt, especially when you have those tricky convenience samples. That sounds promising for people trying to get answers from messy survey data without needing huge probability samples.

Lalam: What they claim is that fine-tuning the LLM on the available data allows it to impute self-reported vote choice under both random and systematic nonresponse, which suggests it's a good strategy for researchers working with non-probability samples or systematic missingness in surveys.

Tom: So, what are the actual results they found? Are these fine-tuned models actually better than the standard tools they compared them to?

Jane: Well, the results show that when representative training data are available, fine-tuned LLMs achieve predictive performance comparable to strong tabular baselines and substantially outperform zero-shot prompting.

Lu: That's interesting because it suggests that if you have good training data, the contextual knowledge from pre-training combined with task-specific tuning helps the model get accurate.

Meng: But what about when the data is biased, like in those small convenience samples? Does that performance hold up?

Jane: Under small and biased convenience samples, fine-tuned LLMs more accurately recover both individual-level vote choice and population-level distributions than zero-shot approaches and often outperform tabular methods.

Tom: That’s a big deal if it holds up under those biased conditions because that’s where most real surveys are happening. So, what does this mean for the practical application of these models?

Paper summary: Lalam: These findings suggest that fine-tuned LLMs can combine sample-specific information with contextual knowledge acquired during pre-training, enabling more robust inference under conditions of sampling bias and missing data.

Jane: It means we might be able to use these tools to impute previously difficult to reconstruct survey data and potentially plan new survey designs where you only need to sample easy-to-reach subpopulations.

Tom: So, just to wrap up this part of the paper, what's the big picture here regarding their methodology?

Lu: The study systematically varied both the size and composition of training data, ranging from large representative samples to small, biased convenience samples, and they compared zero-shot prompting and supervised fine-tuning against established tabular classifiers such as CatBoost.

Meng: And how did they set up the fine-tuning process? That sounds like a lot of setup work on their end.

Jane: They used instruction-tuned models, also known as chat models, where each instruction has a system prompt defining the task and a user prompt with the relevant information for each instance. For example, they used GLES two thousand seventeen post-election cross-section data for this task <ref:2509.25063#pg2,GLES 2017 post-election cross-section>.

Tom: And how did they actually get those vote choices out of the model? Was it just guessing, or was there a specific extraction method?

Lalam: The answer extraction was performed by selecting the highest probability token at the position of the first vote token, following an approach by Argyle et al. on that front.

Lu: They did several experiments to test two main research questions: whether LLMs are more effective in imputing random survey responses than established imputation methods and zero-shot LLMs, and whether they are better when systematic non-response bias is present.

Jane: And they tested this by simulating different types of missingness across several experiments, specifically looking at missing completely at random versus systematically non-responded participants from things like students or specific regions.

Tom: When you look at the results for the random missing data case, what did the comparison between fine-tuned LLMs and those tabular baselines show?

Jane: The results showed that when data are missing completely at random, fine-tuned LLMs match tabular classification methods but clearly outperform their zero-shot capabilities.

Lu: The total variation distance for experiments E1a and E1b indicates that both the baseline classifiers and the fine-tuned models approximate the true distribution of vote choices more closely than the zero-shot models.

Paper summary: Meng: So, in terms of what breaks at scale, is there a concern about how much data you actually need for this fine-tuning?

Lalam: For missing completely at random imputation of closedended items, established classifiers remain competitive and compute-efficient. But when only biased convenience samples are available, fine-tuning small open-source LLMs can recover both individual-level predictions and population level distributions more accurately than zero-shot and often better than tabular methods.

Tom: So the main implication for us in the engineering side is that if we can get those smaller convenience samples, fine-tuning seems like a way to make those models actually useful.

Jane: Exactly, it suggests that these fine-tuned LLMs could facilitate the imputation of previously difficult to reconstruct survey data and pave the way for new planned missing data survey designs.

Lu: The paper notes a limitation there, though: they focused on single-item non-responses, and they also used zero-shot LLMs as baselines without varying the prompts themselves.

Meng: And what about those limitations regarding the data requirements? Does the paper give any numbers on how much complete data you need for stable gains in performance?

Jane: They noted that quantifying the data requirements for stable gains in performance of fine-tuned models is required, which points toward future work needing to be done.

Tom: So, looking at "Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples", the authors are showing that by fine-tuning LLMs on available data, you can impute vote choices even when the sampling is biased or there's missing data.

Jane: They found that while standard classifiers can handle random missing data efficiently, fine-tuned LLMs really shine when you're dealing with biased convenience samples, outperforming both zero-shot models and sometimes even tabular methods in terms of recovering both individual predictions and population distributions.

Lu: The title points to the idea that learning from convenience samples is a viable path for solving survey non-response issues using these models.

Meng: It means that for production systems dealing with messy, real-world survey data, fine-tuning an LLM seems like a much more practical approach than just using off-the-shelf zero-shot predictions.

Lalam: Ultimately, they suggest that combining sample specific information with general knowledge from pre-training allows these models to be more robust under sampling bias and missing data.

Conclusion: Tom: So we've looked at how fine-tuning large language models can help researchers get better guesses about voting choices when they only have messy, biased survey data from things like German elections and those convenience samples.

Jane: Right, so the authors are calling this paper "Learning from Convenience Samples." It basically means they are showing that you don't need massive, perfectly representative samples to make good predictions if you use the right tuning method on what data you actually have.

Tom: Yeah, the authors, Alberto Acerbi and Joseph M. Stubbersfield. They set out to see if fine-tuning an AI model on available survey answers could beat traditional methods when that data is skewed or missing in a specific way.

Jane: The big implication here is that for people doing social science research, this suggests a new path for getting population-level insights even when the sampling process itself has biases baked in.

Tom: It’s about using the AI's existing knowledge from its training to fill in the gaps left by bad data collection or missing responses. It’s not about creating perfect data, it’s about making sense of imperfect data better than we thought possible.

Jane: Exactly, it changes how we think about what kind of survey designs are actually necessary to get valid results for public opinion research.

Tom: This paper also sets up a few limitations, though. The authors admit they focused on single items and didn't test every possible prompt strategy. It’s not a complete solution just yet, but it points toward where the next round of research needs to go.

Jane: So the win is that fine-tuning can pull together sample-specific details with general knowledge from the model, making its answers more robust in those tricky situations.

Tom: The question now becomes how much clean data you actually need to make this work reliably across different scenarios. That’s what we’re looking at next.

Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier, Claudia Wagner

LMU Munich · Munich Center for Machine Learning · GESIS - Leibniz Institute for the Social Sciences · RWTH Aachen University · Complexity Science Hub Vienna

cs.CY, cs.CL

Submitted: 2025-09-29

Updated: 2026-10-16

Comments: Accepted at KONVENS 2026

Code: https://github.com/tobihol/finetuning-survey-imputation

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist The fine-tuning of Large Language Models (LLMs) on available data allows them to impute self-reported vote choice under both random and systematic nonresponse, suggesting they offer a

Key concepts

Zero-shot Prompting: This is using an LLM without any specific training on the task at hand. The model relies solely on its general knowledge to predict answers based only on the instructions given in the prompt. It serves as a baseline comparison to see if specialized fine-tuning provides a meaningful improvement.
Supervised Fine-Tuning (SFT): This involves training an LLM further using a specific, labeled dataset (like survey responses). This process allows the model to learn patterns and nuances specific to the data being used, enabling it to generate more accurate and contextually relevant predictions than a general model.
Biased Convenience Samples: These are small groups of people selected for a survey who do not represent the whole population. The study tested how LLMs perform when trained only on these specific, potentially skewed groups, simulating real-world sampling problems like non-response bias.
Imputation: This is the process of estimating or filling in missing data points in a survey. In this context, the LLM attempts to predict what a respondent would have answered for questions they missed or didn't answer, helping researchers recover complete data sets.

Terminology

Fine-Tuning — 4× in this paper · explained in 5 episodes
Generalization — 1× in this paper · explained in 13 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The fine-tuning of Large Language Models (LLMs) on available data allows them to impute self-reported vote choice under both random and systematic nonresponse, suggesting they offer a promising strategy for researchers working with non-probability samples or systematic missingness in surveys.

How it works

The study investigates using LLMs to simulate respondents via persona-based prompts and fine-tuning them on available data to impute self-reported vote choice under both random and systematic nonresponse, using the German Longitudinal Election Study The researchers compare zero-shot prompting and supervised fine-tuning against tabular classifiers (e.g., CatBoost) and test how different convenience samples (e.g., students) used for fine-tuning affect generalization

The methodology involves several key steps for the experiments

Survey Data Processing involved using the German Longitudinal Election Study (GLES) Cross-Section Cumulation 2009-2017 dataset for our experiments (GLES 2020), which is openly available for academic research
Item Selection Survey items were selected based on factors commonly associated with vote choice in German elections (Schmitt-Beck et al. 2022; Schoen et al. 2017; Klein 2014)
LLM Setup involved using instruction-tuned models (also kown as chat models) where each instruction consists of a system prompt that defines the task and a user prompt that provides the relevant information for a given instance
Supervised Fine-Tuning was conducted on a 40 GB partition of an A100 GPU using low-rank adapters (LoRA) to reduce memory requirements and improve efficiency
Answer Extraction was performed by selecting the highest-probability token at the position of the first vote token, following the approach of Argyle et al. (2023)

Experimental Design and Research Questions

The experiments were designed to address two main research questions

RQ1 was: Are LLMs more effective and robust in imputing random survey responses than established imputation methods and zero-shot LLMs?

RQ2 was: Are LLMs more effective and robust in imputing survey responses than established imputation methods when systematic non-response bias is present?

To test these questions, several experiments were conducted by simulating different types of missingness and evaluating how well different methods can impute the missing answers of the GLES participants

E1 focused on imputing answers that are missing completely at random, comparing fine-tuned LLMs to classifier baselines and zero-shot predictions
E2 explored systematic non-responses by only training on the student population, simulating a convenience sample
E3 simulated systematic non-responses by only training on participants from the state Thuringia, exploring regional bias
E4 simulated systematic non-responses by only training on unemployed participants, motivated by evidence that unemployed individuals are more likely to participate in surveys when monetary incentives are offered

Results and Findings

The results showed that when data are missing completely at random, fine-tuned LLMs match tabular classification methods but clearly outperform their zero-shot capabilities The total variation distance (cf. Figure 4) for E1a and E1b indicates that both the baseline classifiers and the fine-tuned models approximate the true distribution of vote choices more closely than the zero-shot models

Under biased convenience samples, fine-tuning small open-source LLMs (Llama-3 3B and 8B) generally recovers both individual-level predictions and aggregate voteshare distributions more faithfully than zero-shot, often better than tabular methods Specifically for E2, the fine-tuned models retained more of their performance in terms of macro F1 compared to zero-shot models For E3, the 8B Llama fine-tune was able to outperform the classifier baseline (W = 2.40, p =.016 for E3b)

The study found that for missing completely at random imputation of closedended items, established classifiers remain competitive and compute-efficient However, when only biased convenience samples are available, fine-tuning small open-source LLMs can recover both individual-level predictions and populationlevel distributions more accurately than zero-shot and often better than tabular methods This suggests that fine-tuned LLMs could facilitate the imputation of previously difficultto-reconstruct survey data and pave the way for new planned missing-data survey designs, where only easy-to-reach subpopulations need to be sampled

Implications and Limitations

The findings have implications for survey practices

(i) for missing completely at random imputation of closedended items, established classifiers remain competitive and compute-efficient

(ii) when only biased convenience samples are available, fine-tuning small open-source LLMs can recover both individual-level predictions and populationlevel distributions more accurately than zero-shot and often better than tabular methods

Limitations noted include focusing on single-item non-responses, the use of zero-shot LLMs as baselines without varying the prompts, and the case study being limited to one important survey variable, vote choice Future work should investigate whether these findings extend to different variables, surveys, and open-ended responses Additionally, quantifying the data requirements (e.g., the minimum number of complete cases) for stable gains in performance of fine-tuned models is required

Ethical considerations highlight that generated responses may inadvertently misrepresent genuine public sentiment, and while fine-tuning might help to reduce these issues, it does not mitigate them completely The work was supported by the German Research Foundation (DFG, project no. 504226141)

The paper is supported by references such as Alberto Acerbi and Joseph M. Stubbersfield 2023 on Large Language Models Show Human-like Content Biases in Transmission Chain Experiments, James Bisbee et al. 2024 on Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, and Tim Dettmers et al. 2023 on QLoRA: Efficient Finetuning of Quantized LLMs.

--- Page 1 ---

Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study

Abstract Survey researchers face two key challenges: the rising costs of probability samples and missing data (e.g., non-response or attrition), which can undermine inference and increase the use of convenience samples Recent work explores using large language models (LLMs) to simulate respondents via persona-based prompts, often without labeled data We study a more practical setting where partial survey responses exist: we fine-tune LLMs on available data to impute self-reported vote choice under both random and systematic nonresponse, using the German Longitudinal Election Study We compare zero-shot prompting and supervised fine-tuning against tabular classifiers (e.g., CatBoost) and test how different convenience samples (e.g.

Improvements for AI systems

Fine-tuned LLMs can impute vote choices under biased convenience samples more accurately than zero-shot approaches, suggesting a promising strategy for researchers working with non-probability samples or systematic missingness.
Open-source LLMs of size 3B to 8B can recover both individual-level predictions and population-level distributions more faithfully than zero-shot, even when compared to GPT-4o.
The system can be designed to predict vote choice by using a prompt structure that includes a system prompt defining the task, user description providing participant data, and requiring the model to return only a specific label from a predefined set.
For missing completely at random data, fine-tuned LLMs match tabular classifiers and outperform zero-shot approaches when predicting vote choice from GLES 2017 post-election cross-sections.
When facing systematic non-response bias (like training only on students or a specific region), fine-tuned models can show improvements over baselines, for instance, the 3B and 8B Llama fine-tunes achieved higher macro F1 and substantially lower total variation distance than zero-shot models.

Abstract

Survey research increasingly relies on convenience samples as rising costs and nonresponse make representative probability sampling more difficult. This creates a central challenge for public opinion research and electoral studies: how to recover reliable population-level inferences from socially and demographically biased data. While recent work has explored the use of large language models (LLMs) for simulating survey responses, most studies focus on the U.S. population and assume relatively favourable data conditions with limited or random missingness. In this work, we investigate whether LLMs can leverage contextual knowledge about socio-demographic and political patterns to predict vote choice in the German population under sampling constraints. We systematically vary both the size and composition of training data, ranging from large representative samples to small, biased convenience samples, and compare zero-shot prompting and supervised fine-tuning against established tabular classifiers such as CatBoost. Our results show that when representative training data are available, fine-tuned LLMs achieve predictive performance comparable to strong tabular baselines and substantially outperform zero-shot prompting. Under small and biased convenience samples, however, fine-tuned LLMs more accurately recover both individual-level vote choice and population-level distributions than zero-shot approaches and often outperform tabular methods. These findings suggest that fine-tuned LLMs can combine sample-specific information with contextual knowledge acquired during pre-training, enabling more robust inference under conditions of sampling bias and missing data.