On the Role of Citations in Preference Data

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "On the Role of Citations in Preference Data".

Jane: Here's the gist of the paper. The gist The role of citations in preference data for human judges and LLMs within scientific question answering reveals that humans prefer more diverse but fewer overall citations, while LLMs show some stronger citation-related preferences depending on the data and specific models.

Tom: General Setup The study investigates how citation-related features influence human preferences in scientific question answering by drawing on mixed-effects models (MEMs) to compare judgments across two datasets: SciArena and ResearchQA The analysis requires datasets that satisfy three specific conditions: 1.

Jane: Responses must contain citations 2.

Tom: Each query must have multiple responses so that pairwise comparisons are possible 3.

Title and authors: Tom: Let's talk about who wrote this. It’s Yu Hou, Hal Daumé III, Rachel Rudinger, and William Walden from the University of Maryland and Johns Hopkins University.

Jane: It's a team that seems to have a solid background in NLP and machine learning research, which is exactly what you need when you're looking at how models process complex information like citations.

Tom: Yeah, they’re researchers who are using mixed effects models to interrogate these questions about citation features influencing pairwise judgments on scientific question answering responses.

Lu: They set up experiments comparing human preferences against judges from four different open-weight LLMs: Llama3 point 3-70B, Gemma3-27B, Qwen3-30B, and Skywork-Critic-70B.

Meng: Comparing them to humans is smart because it shows where the AI's internal decision process starts to diverge from what a person would naturally prefer.

Lalam: The key finding here is that LLMs show some stronger citation preferences compared to humans, but those preferences aren't universal; they depend on the specific data and the model being tested.

The paper's summary: Tom: So, what’s the core of what this paper says? It investigates how citation-related features affect preference data when judging scientific answers.

Jane: Simply put, they found two main things: first, human judges prefer responses that have a wider variety of cited sources but don't use as many citations in total.

Tom: That’s right. And second, the LLM experiments showed that these citation features influence their preferences differently than they do for humans, which suggests potential biases in how we train reward models for these systems.

Lu: They used specific predictors like citation diversity, which is measured by the Shannon diversity index—basically how varied the sources are—and citation count.

Meng: And they also looked at things specific to ResearchQA, like citation mismatch, which measures when an in-text citation doesn't match what’s in the reference list.

Lalam: The paper highlights that for ResearchQA human judges prefer more diverse citations but fewer total citations, and the citation mismatch feature significantly correlates with a response's probability of being chosen.

The paper's improvements: Tom: The authors suggest a few ways to improve how we use this preference data. They think we should design better instructions for annotators so they understand cited content more easily before and during the collection process.

Jane: That makes sense because if the annotators don't clearly know what’s being cited, their judgments might be based on assumptions instead of actual content quality.

Lu: For LLM judges specifically, they suggest incorporating external tools, like giving them access to cited documents through external tools, to help them better approximate human evaluations for long-form responses.

Meng: Having the AI look up the source directly should help bridge that gap where the AI only sees text without knowing what’s actually behind it.

Lalam: They also suggest incorporating a bias check based on response content uniqueness for LLM judges because they found that this predictor shows much stronger correlations across all LLM judges than those of human judges.

Conclusion: Tom: So, to wrap up the paper "On the Role of Citations in Preference Data," humans prefer more diverse citations but fewer overall citations, and LLMs process this information in a fundamentally different way.

Jane: That suggests that while we can use this data to train reward models, we need to be careful about how we interpret those preferences because the mechanics are different for the AI.

Lu: The paper points toward using these findings to inform reward model training by generating responses that satisfy both diversity and conciseness preferences at the same time.

Meng: I’m thinking if we want reliable AI outputs, maybe we need to focus on making sure our judging systems are sensitive to those citation details, like what they suggested about content uniqueness.

Lalam: This work really contributes to understanding the role citations play in preference data from both human and LLM judges, showing that even when comparing outputs without the original sources, these citation features matter a lot.