FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays

arXiv:2606.06271 · cs.CL, cs.HC · Submitted 2026-06-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays. Jane, what’s the big picture here?

Jane: Well, basically, it’s about comparing how human writing instructors give feedback to argumentative essays versus what four different frontier Large Language Models produce when they are given the same instructions. It sets up a system to see where the human and model feedback lines up and where they totally drift apart in terms of what's important in revision.

Lu: I think it’s significant because for writing research, we always talk about three key things: goal-orientation, anchoring to specific sentences, and prioritization. This paper is the first one systematically comparing how instructors and models handle those exact three dimensions across a whole dataset.

Meng: So it’s not just about whether the feedback is good or bad; it’s about *how* they focus on the essay structure itself, right? Like, which parts of the argument are they actually hitting when they give advice?

Tom: Exactly. The authors built this dataset called FOXGLOVE, which has a total of two thousand three hundred forty comments—six hundred ninety-six from human instructors and one thousand six hundred forty-four from four different AI models—all using the same set of rules <ref:2606.06271#pg1>.

Jane: And what they found is that while both groups generally agree on which argumentative goals are important across essays and positions, they really diverge when it comes to *which specific sentences* they pick to comment on and how urgent those comments are.

Lu: I also saw something interesting about the models themselves; the paper notes that LLMs tend to write feedback that is more complex than instructors do, and they use fewer questions overall.

Meng: That’s a big point for deployment because complexity can be good, but if it’s not actionable or focused on what actually needs fixing, it just adds noise at scale.

Tom: And the quality ratings are a bit nuanced; LLM feedback often gets higher scores on most quality dimensions from the instructors, but the paper immediately points out that this advantage is mostly because their comments are longer.

Jane: So even when they agree on what’s important, we have to be careful about trusting the model just because it wrote a longer piece of text. That leads us right into how they actually constructed this comparison.

The paper's summary: Tom: Now that we know what FOXGLOVE is, let's talk about the core methodology described in the paper. Basically, they’re using argumentative essays from a corpus called PERSUADE two point zero—which consists of high-school student writing—and annotating every sentence with five specific goals: Position, Claim, Evidence, Counterclaim, and Rebuttal <ref:2606.06271#pg2>.

Jane: That sentence-level labeling is crucial because it lets them measure feedback directly against those five rhetorical goals for each part of the essay. It’s not just a general comment; it’s tied to a specific argumentative task.

Lu: They also established a very specific feedback schema that everyone, both humans and models, had to follow. Every piece of feedback has three parts: sentence-level comments anchored to the text, an urgency ranking where you pick the top three most important ones—ranked one being most urgent—and a global comment summarizing how the essay fits the goals.

Meng: That shared protocol is what makes it a true comparison; if they weren't all following the same rules, we wouldn't know if their differences were due to different training or just different ways of thinking about the task.

Tom: Right, and this setup allowed them to systematically compare where instructors and models align on those goals versus where they diverge on sentence selection and prioritization.

Jane: The summary emphasizes that even though the distribution of feedback across goals seems similar between humans and AI, the specific choice of text—the anchoring—and the urgency ranking are where the real differences emerge.

Lu: It’s interesting because they found that models tend to write more complex feedback and use fewer questions than instructors when we look at those sentence-level comments.

Meng: That makes sense if the model is trying to be thorough, but Tom mentioned earlier that this length can actually hurt quality scores unless it translates into better specificity.

Tom: It does. The paper sets up this comparison so we can see if that extra length in the AI feedback actually leads to better results or just looks good on paper because it’s longer.

The paper's improvements: Tom: So, the authors didn't just stop at showing the differences; they suggested ways to make their work better, and those improvements are pretty interesting for how we use this kind of data.

Jane: The first improvement they suggest is to fix the issue with models plateauing on certain types of text spans. They found that GPT-five point two and Qwen3-Next 80B kept failing consistently on a few specific sentence highlighting cases, which suggests we need better refinement loops than just simple resamplings <ref:2606.06271#pg2>.

Lu: And the second improvement is directly addressing the length bias we talked about earlier. They pointed out that longer comments correlate with higher scores in specificity and actionability for both groups, so they propose a way to control the output length to be closer to a human average of around sixty-four words <ref:2606.06271#pg1>.

Meng: Controlling the length is smart because it removes that confounding variable where the models look better just because they write more text. If we force them toward that average, we get a fairer comparison of quality versus quantity.

Tom: And the third suggestion is about testing if eliciting more varied feedback actually helps usefulness, rather than just optimizing for expert ratings alone. They want to see if variation can be a strength or just noise.

Jane: That’s a really important direction for future research because it moves beyond just getting the highest possible score and asks whether different styles of feedback actually serve different revision needs.

Lu: I think those suggestions give us concrete protocols we can use in the future to test if we can actually guide models toward producing feedback that is more focused on what matters for the student’s learning process.

Meng: If they get a method that forces length control, that’s something my team could start prototyping right away to make sure our deployed models aren't just outputting verbose nonsense.

Conclusion: Tom: So wrapping up the FOXGLOVE paper, the main implication is that we can finally have a systematic way to compare where human and AI feedback agree and disagree on the core elements of revision: what goals to target, which specific text spans matter, and how urgent those comments are.

Jane: It shows that while AI can generate comprehensive feedback in terms of length, human judgment still remains essential for deciding the precise focus needed for effective revision.

Lu: The paper gives us reusable protocols that could be used in writing-intensive contexts to test if we can connect rated quality directly to actual improvements in student writing outcomes.

Meng: For deployment, it suggests that we need methods to manage the output style so we aren't just relying on length as a proxy for usefulness when integrating AI feedback into workflows.

Lalam: From my perspective, FOXGLOVE confirms that the models are very good at covering all five goals in their global comments, but they still struggle with the fine-grained anchoring to the exact sentences that matter for a specific claim or evidence point.

University of Illinois Urbana-Champaign

cs.CL, cs.HC

Submitted: 2026-06-04

Updated: 2026-10-19

Code: https://github.com/yijunliu23/foxglove_data_

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 92/100

The gist: The gist: FOXGLOVE introduces a dataset comparing feedback from trained human instructors and four frontier Large Language Models (LLMs) on argumentative essays to systematically analyze where human

Key concepts

FOXGLOVE Dataset
A dataset containing 2,340 comments—696 from human instructors and 1,644 from four LLMs—based on argumentative essays. It is used to systematically compare how humans and models give feedback across goals, sentence selection, and prioritization.
Feedback Schema
The structured way feedback is given by both humans and models. It includes three parts: sentence-level comments addressing specific essay goals (like Position or Evidence), an urgency ranking of the top three comments, and a single global comment summarizing the essay's alignment with goals.
Goal-Orientation
The focus of feedback on what aspect of the argument needs improvement. The study analyzed how instructors and LLMs distribute their feedback across five specific goals: Position, Claim, Evidence, Counterclaim, and Rebuttal. This shows where human and model advice agree or disagree on which part of the essay is most important to fix.
Anchoring to Specific Sentences
The practice of pointing out exact sentences in an essay that require revision. The analysis showed that instructors and LLMs diverge significantly on which specific sentences they choose to highlight, indicating a difference in how they pinpoint areas for improvement.

Terminology

Summary

The gist: FOXGLOVE introduces a dataset comparing feedback from trained human instructors and four frontier Large Language Models (LLMs) on argumentative essays to systematically analyze where human and model feedback align, diverge, and differ regarding goal-orientation, anchoring to specific sentences, and prioritization.

Dataset Construction

FOXGLOVE is a dataset of 2,340 comments consisting of 696 feedback comments from trained human instructors paired with 1,644 comments generated by four frontier LLMs under a shared protocol (Page 2). The dataset is built from essays sampled from the PERSUADE 2.0 corpus, which consists of argumentative essays written by high-school students (Page 3). Each essay in the dataset averages approximately 423 words and is annotated at the sentence level with discourse element labels that serve as five goal labels: Position, Claim, Evidence, Counterclaim, and Rebuttal (Page 3).

Feedback Schema

Every feedback giver produces feedback for each essay according to a shared schema consisting of three components (Page 3):

  1. Sentence-level feedback: A free-text comment anchored to one or more contiguous sentences in the essay, addressing one of the five argumentative goals (Page 3). If multiple goals are addressed on the same span, multiple comments are produced (Page 3).

  2. Urgency ranking: After all sentence-level comments are produced, the feedback giver selects and ranks the top three most important comments in order of urgency (1 being most urgent) (Page 3).

  3. Global comment: A single free-text comment per essay synthesizing how the essay aligns with the stated argumentative goals (Page 3).

Analysis of Divergence

The analysis reveals that while instructors and LLMs distribute feedback similarly across argumentative goals and essay positions, they diverge on specific sentence selection and urgency rankings (Page 1). Models tend to write more complex feedback and use fewer questions than instructors (Page 1). Furthermore, LLM feedback receives higher ratings on most dimensions of quality as rated by instructors, but this advantage is largely attributed to the lengthier comments generated by the models (Page 1).

Content and Language Differences

Analysis of comment language shows distinct differences between human and model feedback (Page 4.2). LLM comments are generally longer than instructor comments, averaging 83.64 words compared to an instructor average of 34.70 words for sentence-level comments (Page 5). Readability measures indicate that LLM feedback is significantly less readable, with Flesch–Kincaid grades around 14.4 for LLMs versus grade 8.5 for instructors (Page 6). Instructors use more first-person pronouns and question marks per 100 words compared to LLMs, suggesting instructor feedback is more conversational (Page 10).

Quality Ratings and Length Bias

Independent expert quality ratings across seven dimensions were collected from two trained writing instructors on a subset of feedback comments (Page 5). LLMs generally receive higher scores in all dimensions except for tone (Page 8). A key finding is that the apparent human-model rating gap is substantially driven by comment length, as longer comments correlate with higher scores in specificity and actionability for both groups (Page 12).

Conclusion and Implications

FOXGLOVE enables a systematic comparison of where human and LLM feedback align, diverge, and differ across goals, anchoring points, and prioritization (Page 1). The dataset provides reusable protocols to support future studies connecting rated quality to revision outcomes (Page 6.3). The findings suggest that while LLMs can produce feedback that is comprehensive in length, human judgment remains important for personalization and prioritization (Page 2).

--- Page 1 ---

FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback

Feedback from Experts and LLMs on Argumentative Essays

Yijun Liu, Yifan Song, John Gallagher, Sarah Sterman, Tal August

University of Illinois Urbana-Champaign

Urbana, IL, USA

Abstract

While large language models (LLMs) are increasingly used to generate writing feedback, there remains no systematic comparison of LLM and expert feedback on the dimensions that writing research identifies as central to revision: goal-orientation, anchoring to specific sentences, and prioritization (Page 1). We introduce FOXGLOVE, a dataset of 696 feedback comments written by trained writing instructors on 69 twelfth-grade argumentative essays, paired with 1,644 comments generated from four frontier LLMs under a shared protocol, totaling 2,340 comments (Page 1). We provide expert quality ratings on a subset of both instructor and LLM comments (Page 1). We find that instructors and LLMs distribute feedback similarly across goals and essay positions, yet instructors and models diverge on the specific sentences on which to provide feedback (Page 1). Additionally, we find that models tend to write more complex feedback and use fewer questions than instructors (Page 1). LLM feedback also receives higher ratings on most dimensions of quality, as rated by instructors, but much of this advantage appears to be attributable to lengthier comments (Page 1). FOXGLOVE enables systematic comparison of where human and LLM feedback align, diverge, and differ (Page 1).

1 Introduction

Writing feedback shapes how students develop as writers (Mah et al., 2025), and that feedback is increasingly written by machines (Page 1). Large language models (LLMs) can now produce feedback that engages with multi-dimensional analytic assessments including student writings’ richness, visualization, interactivity, and personalization (Guerraoui et al., 2023) (Page 1). Such feedback has been integrated into classroom workflows alongside teachers and teaching assistants (Lu et al., 2026; Pahi et al., 2024; Zhang and Jiang, 2026) (Page 1). However, for writers and instructors to rely on generated feedback, there needs to be effective ways of determining how LLM feedback differs from the feedback instructors traditionally give (Page 1). Decades of writing research converge on what effective feedback should do: it engages the rhetorical goals a writer is pursuing (Flower and Hayes, 1981) and points to the specific text where revision should occur (Churchill et al., 2000; Weng and Gennari, 2004; Zyto et al. 2012) while prioritizing the most important revisions (Sommers, 1982) (Page 1). Whether human and model feedback align with one another on these criteria is unknown (Page 1). This motivates our central question: when humans and LLMs respond to the same essays under the same instructions, do they agree on where revision is needed, which argumentative goal should be addressed, and what feedback is the most urgent? (Page 1). While past work has qualitatively described these differences (Mah et al., 2025), answering this question at scale requires data that is presently unavailable (Table 1) (Page 2). We address this gap by introducing FOXGLOVE, a dataset we collect with anchored, goallabeled feedback comments from trained human instructors (696 comments) and four models (1,644 comments from GPT-5.2, Claude Sonnet 4.5, Llama 3.3 70B, and Qwen3-Next 80B) produced under a shared protocol (Page 2).

Dataset Anchored Feedback

Labels

Human

Feedback

LLM

Feedback

Urgency

Rank

Human-Rated: Exposía (Zyska et al., 2026) Peer & Instructor; Coyne et al. (2025) English Teacher; LEAF++ (Misgna et al., 2026) Online Forum; LEAF (Behzad et al., 2024b) Online Forum; SEFL (Zhang et al., 2026) X X X; Mah et al. (2025) X English Teacher; Pilan et al. (2020) X Tutor; ArgRewrite v2 (Kashefi et al., 2022) Trained Expert X X X; Rashkin et al. (2025) X X X; FOXGLOVE Instructor Table: Comparison of multiple datasets on writing feedback (Page 3).

Table 1: Comparison of multiple datasets on writing feedback.

We address this gap by introducing FOXGLOVE, a dataset we collect with anchored, goallabeled feedback comments (Figure 1) from trained human instructors (696 comments) and four models (1,644 comments from GPT-5.2, Claude Sonnet 4.5, Llama 3.3 70B, and Qwen3-Next 80B) produced under a shared protocol (Page 2). With the dataset, we systematically compare instructor and LLM feedback (Page 2). Our paper makes the following contributions:

• A parallel feedback dataset of 2,340 comments on 69 twelfth grade argumentative essays (Crossley et al., 2024), produced by 14 expert writing instructors and a shared instructional protocol for eliciting model feedback in the same form (Page 2).

Improvements for AI systems

  1. Improve LLM feedback generation by incorporating stricter adherence to schema validation, as models plateaued on GPT-5.2 and Qwen3-Next 80B through v8, both models continued to fail deterministically on the same handful of span-highlighting cases. This suggests developing iterative refinement loops that go beyond simple resamplings to address persistent failure modes identified in Table 5.

  2. Enhance LLM feedback quality by explicitly mitigating the bias toward length, since longer comments simply look more thorough for human raters and Length was a significant positive predictor of every dimension except tone. Implement a length-control mechanism that forces models to generate feedback closer to the average human comment length of approximately 64 words, which is where both sides' ratings converge.

  3. Develop a comparative evaluation framework for LLM feedback that tests whether eliciting more varied, reader-specific feedback improves usefulness, moving beyond optimizing solely toward expert-rated quality. This allows future systems to be tested on whether variation can be a strength or merely reflect[s] inconsistency or noise.

Abstract

While large language models (LLMs) are increasingly used to generate writing feedback, there remains no systematic comparison of LLM and expert feedback on the dimensions that writing research identifies as central to revision: goal-orientation, anchoring to specific sentences, and prioritization. We introduce FOXGLOVE, a dataset of 696 feedback comments written by trained writing instructors on 69 twelfth-grade argumentative essays, paired with 1,644 comments generated from four frontier LLMs under a shared protocol, totaling 2,340 comments. We provide expert quality ratings on a subset of both instructor and LLM comments. We find that instructors and LLMs distribute feedback similarly across goals and essay positions, yet instructors and models diverge on the specific sentences on which to provide feedback. Additionally, we find that models tend to write more complex feedback and use fewer questions than instructors. LLM feedback also receives higher ratings on most dimensions of quality, as rated by instructors, but much of this advantage appears to be attributable to lengthier comments. FOXGLOVE enables systematic comparison of where human and LLM feedback align, diverge, and differ.

Sources

Related papers