Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Learning from Convenience Samples".

Jane: Here's the gist of the paper. The gist The fine-tuning of Large Language Models (LLMs) on available data allows them to impute self-reported vote choice under both random and systematic nonresponse, suggesting they offer a promising strategy for researchers working with non-probability samples or systematic missingness in surveys.

Tom: How it works The study investigates using LLMs to simulate respondents via persona-based prompts and fine-tuning them on available data to impute self-reported vote choice under both random and systematic nonresponse, using the German Longitudinal Election Study The researchers compare zero-shot prompting and supervised fine-tuning against tabular classifiers (e.g., CatBoost) and test how different convenience samples (e.g., students) used for fine-tuning affect generalization The methodology involves several key steps for the experiments 1.

Jane: Survey Data Processing involved using the German Longitudinal Election Study (GLES) Cross-Section Cumulation 2009-2017 dataset for our experiments (GLES 2020), which is openly available for academic research 2.

Tom: Item Selection Survey items were selected based on factors commonly associated with vote choice in German elections (Schmitt-Beck et al.

Paper summary: Tom: So we're talking about this paper called "Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples". This paper tackles a big problem in survey research: the rising costs of probability samples and all that missing data, like non-response or people dropping out.

Jane: Right, so the core idea here is using large language models, or LLMs, to simulate responses when you don't have perfect data. The authors are looking at a more real-world scenario where you have partial survey answers and trying to figure out vote choice in Germany using the German Longitudinal Election Study.

Lu: They set out to compare two main approaches: zero-shot prompting and supervised fine-tuning against established methods like CatBoost classifiers. The paper claims that they can recover reliable population-level inferences even when dealing with socially or demographically biased data and missingness.

Meng: Okay, so the claim is that fine-tuned LLMs can do better than just using a pre-trained model with a simple prompt, especially when you have those tricky convenience samples. That sounds promising for people trying to get answers from messy survey data without needing huge probability samples.

Lalam: What they claim is that fine-tuning the LLM on the available data allows it to impute self-reported vote choice under both random and systematic nonresponse, which suggests it's a good strategy for researchers working with non-probability samples or systematic missingness in surveys.

Tom: So, what are the actual results they found? Are these fine-tuned models actually better than the standard tools they compared them to?

Jane: Well, the results show that when representative training data are available, fine-tuned LLMs achieve predictive performance comparable to strong tabular baselines and substantially outperform zero-shot prompting.

Lu: That's interesting because it suggests that if you have good training data, the contextual knowledge from pre-training combined with task-specific tuning helps the model get accurate.

Meng: But what about when the data is biased, like in those small convenience samples? Does that performance hold up?

Jane: Under small and biased convenience samples, fine-tuned LLMs more accurately recover both individual-level vote choice and population-level distributions than zero-shot approaches and often outperform tabular methods.

Tom: That’s a big deal if it holds up under those biased conditions because that’s where most real surveys are happening. So, what does this mean for the practical application of these models?

Paper summary: Lalam: These findings suggest that fine-tuned LLMs can combine sample-specific information with contextual knowledge acquired during pre-training, enabling more robust inference under conditions of sampling bias and missing data.

Jane: It means we might be able to use these tools to impute previously difficult to reconstruct survey data and potentially plan new survey designs where you only need to sample easy-to-reach subpopulations.

Tom: So, just to wrap up this part of the paper, what's the big picture here regarding their methodology?

Lu: The study systematically varied both the size and composition of training data, ranging from large representative samples to small, biased convenience samples, and they compared zero-shot prompting and supervised fine-tuning against established tabular classifiers such as CatBoost.

Meng: And how did they set up the fine-tuning process? That sounds like a lot of setup work on their end.

Jane: They used instruction-tuned models, also known as chat models, where each instruction has a system prompt defining the task and a user prompt with the relevant information for each instance. For example, they used GLES two thousand seventeen post-election cross-section data for this task <ref:2509.25063#pg2,GLES 2017 post-election cross-section>.

Tom: And how did they actually get those vote choices out of the model? Was it just guessing, or was there a specific extraction method?

Lalam: The answer extraction was performed by selecting the highest probability token at the position of the first vote token, following an approach by Argyle et al. on that front.

Lu: They did several experiments to test two main research questions: whether LLMs are more effective in imputing random survey responses than established imputation methods and zero-shot LLMs, and whether they are better when systematic non-response bias is present.

Jane: And they tested this by simulating different types of missingness across several experiments, specifically looking at missing completely at random versus systematically non-responded participants from things like students or specific regions.

Tom: When you look at the results for the random missing data case, what did the comparison between fine-tuned LLMs and those tabular baselines show?

Jane: The results showed that when data are missing completely at random, fine-tuned LLMs match tabular classification methods but clearly outperform their zero-shot capabilities.

Lu: The total variation distance for experiments E1a and E1b indicates that both the baseline classifiers and the fine-tuned models approximate the true distribution of vote choices more closely than the zero-shot models.

Paper summary: Meng: So, in terms of what breaks at scale, is there a concern about how much data you actually need for this fine-tuning?

Lalam: For missing completely at random imputation of closedended items, established classifiers remain competitive and compute-efficient. But when only biased convenience samples are available, fine-tuning small open-source LLMs can recover both individual-level predictions and population level distributions more accurately than zero-shot and often better than tabular methods.

Tom: So the main implication for us in the engineering side is that if we can get those smaller convenience samples, fine-tuning seems like a way to make those models actually useful.

Jane: Exactly, it suggests that these fine-tuned LLMs could facilitate the imputation of previously difficult to reconstruct survey data and pave the way for new planned missing data survey designs.

Lu: The paper notes a limitation there, though: they focused on single-item non-responses, and they also used zero-shot LLMs as baselines without varying the prompts themselves.

Meng: And what about those limitations regarding the data requirements? Does the paper give any numbers on how much complete data you need for stable gains in performance?

Jane: They noted that quantifying the data requirements for stable gains in performance of fine-tuned models is required, which points toward future work needing to be done.

Tom: So, looking at "Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples", the authors are showing that by fine-tuning LLMs on available data, you can impute vote choices even when the sampling is biased or there's missing data.

Jane: They found that while standard classifiers can handle random missing data efficiently, fine-tuned LLMs really shine when you're dealing with biased convenience samples, outperforming both zero-shot models and sometimes even tabular methods in terms of recovering both individual predictions and population distributions.

Lu: The title points to the idea that learning from convenience samples is a viable path for solving survey non-response issues using these models.

Meng: It means that for production systems dealing with messy, real-world survey data, fine-tuning an LLM seems like a much more practical approach than just using off-the-shelf zero-shot predictions.

Lalam: Ultimately, they suggest that combining sample specific information with general knowledge from pre-training allows these models to be more robust under sampling bias and missing data.

Conclusion: Tom: So we've looked at how fine-tuning large language models can help researchers get better guesses about voting choices when they only have messy, biased survey data from things like German elections and those convenience samples.

Jane: Right, so the authors are calling this paper "Learning from Convenience Samples." It basically means they are showing that you don't need massive, perfectly representative samples to make good predictions if you use the right tuning method on what data you actually have.

Tom: Yeah, the authors, Alberto Acerbi and Joseph M. Stubbersfield. They set out to see if fine-tuning an AI model on available survey answers could beat traditional methods when that data is skewed or missing in a specific way.

Jane: The big implication here is that for people doing social science research, this suggests a new path for getting population-level insights even when the sampling process itself has biases baked in.

Tom: It’s about using the AI's existing knowledge from its training to fill in the gaps left by bad data collection or missing responses. It’s not about creating perfect data, it’s about making sense of imperfect data better than we thought possible.

Jane: Exactly, it changes how we think about what kind of survey designs are actually necessary to get valid results for public opinion research.

Tom: This paper also sets up a few limitations, though. The authors admit they focused on single items and didn't test every possible prompt strategy. It’s not a complete solution just yet, but it points toward where the next round of research needs to go.

Jane: So the win is that fine-tuning can pull together sample-specific details with general knowledge from the model, making its answers more robust in those tricky situations.

Tom: The question now becomes how much clean data you actually need to make this work reliably across different scenarios. That’s what we’re looking at next.