Assessment of Generative Named Entity Recognition in the Era of Large Language Models

arXiv:2601.17898 · cs.CL · Submitted 2026-01-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Assessment of Generative Named Entity Recognition in the Era of Large Language Models".

Jane: The gist Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs),

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about this paper now, "Assessment of Generative Named Entity Recognition in the Era of Large Language Models." It looks like they are trying to see where generative models fit in with older ways of doing named entity recognition.

Jane: Exactly. Think about how we used to do NER, which was mostly sequence labeling, and now it’s shifting into a generative area because of these large language models we're using today.

Lu: They’re systematically evaluating open-source LLMs on both flat and nested NER tasks to look at the performance gaps, the effect of output formats, whether they rely on memorization, and if their general capabilities stay intact after fine-tuning.

Meng: So what they found is that when you use parameter-efficient fine-tuning alongside structured formats like inline bracketed or XML, these open-source LLMs actually get performance competitive with traditional encoder-based models and even surpass decoder-based LLMs using in-context learning techniques.

Tom: That’s a big point. What they found is that the NER capability of these LLMs comes from following instructions and their generative power, not just memorizing entity-label pairs from their training data.

Jane: And they showed that applying NER instruction tuning doesn't really hurt the general capabilities of the LLMs at all; in fact, it actually improves performance on datasets like DROP by twenty-five point five zero to forty-five point three two F1 points because it enhances how the model understands entities >

Lu: Now, let’s talk about output formats because that seems super important for this generative approach. They found that the inline bracketed and inline XML formats achieve the best average performance across all LLMs, with average F1 scores of ninety-one point nine one/eighty-two point eight five and ninety-one point seven one/eighty-two point six seven on flat and nested NER > <ref:2601.17898#pg3>

Meng: And they also noted that these inline formats consistently outperform the JSON-based formats, which scored around ninety point seven eight/eighty-one point six nine, ninety-one point one seven/eighty point zero seven, and even a lower score of fifty-three point nine one/thirty-four point five three >

Tom: So it seems that clear, structured outputs that look like natural language are much better for helping the LLMs understand and generate accurate entity information compared to formats that require finishing the whole thing at once with precise character-offset calculations >

Jane: That makes sense. It suggests that formats requiring you to conclude all the information at one time and calculate those precise character offsets aren't suited for how LLMs naturally generate text >

Title and authors: Lu: Well, this means well-designed output formats enable LLMs to match or even surpass some traditional fine-tuning methods and closed-source models >

Tom: Moving on, they looked into whether these LLMs rely on memorization of entity labels. The investigation showed that the capability comes mainly from instruction-following and text generation rather than just memorizing entity-label correlations from training corpora >

Jane: Error analysis indicated that LLMs tend to generate the wrong types, which they attribute to a misalignment between their rich intrinsic knowledge and the detailed annotation guidelines of certain datasets >

Meng: But the study concludes that LLMs don't actually rely on entity-label memorization in generative NER >

Tom: It’s pretty interesting because it means we can focus on instruction tuning instead of just trying to memorize every possible label combination, right? And they showed that performance on tasks like DROP improves by twenty-five point five zero to forty-five point three two F1 points with this instruction tuning >

Jane: And what about the preservation of the LLMs' original general capabilities? They found they retain most of their original capabilities under parameter-efficient fine-tuning for NER tasks, with performance staying stable or even decreasing slightly while improving significantly on benchmarks like DROP by twenty-five point five zero to forty-five point three two F1 points >

Lu: The core capabilities of the LLMs remain largely intact after they undergo this lightweight fine-tuning for the NER task >

Tom: So, what’s next? The paper does point out some areas for improvement in future work. They suggest extending the investigation to reasoning models with long chain-of-thought output, which would give us an advantage by tracing entity labels through that reasoning chain and enhancing interpretability >

Jane: And they also mentioned that reinforcement learning based approaches, like PPO, DPO, and GRPO, have been shown to surpass traditional supervised fine-tuning in terms of eliciting model capability and generalization >

Meng: They just didn't explore those reinforcement learning approaches in this specific work >

Tom: Yeah, so it’s an open door for future research then. It confirms that generative NER is a viable and flexible approach that could gradually supplement traditional sequence labeling methods >

Jane: It really does show that with properly constructed prompts, generative NER using open-source LLMs can achieve highly competitive performance by leveraging their strong native language understanding capabilities >

Title and authors: Lu: To wrap things up for now, they also showed that LLaMA3 point 1-8B achieves ninety-three point five zero and ninety-three point eight three on CoNLL2003, surpassing all pre-trained NER baselines with the inline bracketed format >

Tom: That’s a high number for a performance metric, but they also noted that the best LLM result was only four point zero two F1 points lower than the best specialized model BERTMRC and actually fell below other traditional baselines in some low-resource domains >

Jane: So while it shows strong performance, it’s important to remember that in low-resource areas, specialized models can still have an edge over the largest LLMs >

Lu: Overall, the average F1 on flat NER datasets for open-source LLMs is ninety-three point five nine and nested NER datasets is ninety-three point two four > <ref:2601.17898#pg3>

Meng: The best results they showed are bolded in Table three and Table four with the overall average F1 scores for open-source LLMs on flat NER datasets being ninety-three point five nine and on nested NER datasets being ninety-three point two five >

Tom: It sounds like generative NER with LLMs is a promising, user-friendly alternative to traditional methods because you don't have to build those complex sequence labeling pipelines from scratch >

Jane: It’s a very practical idea for how we can approach entity recognition in the future because it leverages the language understanding power we already have in these models >

Lu: And they released their data and code at github.com/szutera/LLMs4NER, which is helpful for others to build on this work >

Meng: I think having that open source means more people can experiment with these results and see how they translate practically into different applications >

Tom: We’ve covered the main points of "Assessment of Generative Named Entity Recognition in the Era of Large Language Models," which is a comprehensive assessment of generative NER using open-source LLMs >

Jane: It really highlights that instruction following and structured output formats are key drivers for success in this new way to do named entity recognition >

Lu: We’ve seen how parameter-efficient fine-tuning keeps general capabilities intact while boosting performance on benchmarks like DROP by twenty-five point five zero to forty-five point three two F1 points >

Meng: And we know that the structure of the output, specifically inline bracketed or XML, makes a real difference for accuracy >

Tom: That’s what we were discussing, and it sets us up perfectly for looking at how these models handle different kinds of data structures next.

The paper's summary: Tom: So, to recap, this paper is taking named entity recognition out of just being some old sequence labeling thing and making it generative using these massive language models we're seeing now.

Jane: Right, they’re checking how these open-source language models actually perform on both simple and complex recognition tasks to see where the gaps are.

Tom: They found that when you fine-tune these open-source models smartly, using methods that don't mess up their general intelligence too much, they can actually keep performance competitive with older, traditional models.

Jane: And the key part is how you structure the output; they showed that formats like inline XML or bracketed text give them the best results overall.

Tom: It means for anyone trying to use these tools for entity extraction, a clear format that looks like normal language is way better than those super precise character-offset JSON formats.

Jane: That makes sense because it lets the model focus on the meaning of the text instead of just worrying about hitting exact positions in a specific format.

Tom: They also looked at whether these models are just memorizing labels, and they concluded that no, their ability comes from following instructions and generating text >

Jane: Which is important because it means we don't have to worry about them just recalling a list of pairs from the training data; they’re actually learning to follow those prompts.

Tom: And that instruction tuning isn't hurting their general skills at all, in fact, it boosts performance on tasks like DROP by nearly fifty points >

Jane: So, for someone listening to this show, what does this mean is that we can use these open-source language models for entity extraction more reliably if we tell them how to do the job through clear instructions.

Tom: It's a pretty practical idea for how we can approach entity recognition in the future because you don't have to build those super complex sequence labeling pipelines from scratch >

Jane: Exactly, it leverages the language understanding power we already have in these models and makes it more accessible than some of those older methods.

Tom: But they did point out a few things they didn't fully explore yet, like looking at reasoning models that use long chain-of-thought output to trace labels through complex steps >

Jane: Yeah, and they mentioned that reinforcement learning approaches have shown better results in eliciting model capability than just standard fine-tuning methods >

Tom: That opens up a lot of doors for future research, especially with those advanced reasoning models.

Jane: It's exciting because it shows that generative NER is a flexible path that can supplement traditional sequence labeling without losing the core intelligence of the language model itself.

The paper's improvements: Tom: So, we just talked about how they figured out that structured output formats like XML actually give these open-source models better scores than JSON >

Jane: Exactly, because that's what helps them understand and generate the entity information more accurately, which is a big deal for real-world use.

Tom: The paper suggests that to push this further, we should look at reasoning models with long chain-of-thought output >

Jane: That means tracing the entity labels through a step-by-step thinking process, which would make the results way more interpretable for us.

Tom: And they also mentioned reinforcement learning approaches like PPO and DPO as something that can surpass traditional supervised fine-tuning methods in terms of making models more capable >

Jane: That's interesting because they didn't actually test those reinforcement learning ideas in this specific work, so it leaves a lot of room for future experiments.

Tom: It suggests that the next step is to really see if these instruction-following and generative powers can be amplified by those more complex reasoning structures >

Jane: And I think that’s where we need to keep our eyes on things like how models handle very long, multi-step instructions for extraction tasks.

Tom: It confirms that using these open-source language models with the right structure is a solid starting point for making NER more user-friendly >

Jane: And it shows that even with just lightweight fine-tuning, we can keep those strong general capabilities intact while still getting better results on specific tasks like entity recognition.

Tom: So, what this means for us is we don't have to overhaul our entire system just to get better entity extraction; a small tweak and a good format can do the heavy lifting >

Jane: It’s practical because it lets us leverage the power of these models without needing massive amounts of specialized data every time.

Tom: We’ve seen how much performance can jump, like on DROP datasets improving by almost twenty-five points with just instruction tuning, so those are numbers we can use >

Jane: And I think that means for everyday applications, whether it's summarizing documents or pulling specific details from text, these models become a much more viable tool.

Tom: We've covered the main ideas and what the authors suggest to do next with reasoning and reinforcement learning >

Conclusion: Tom: So we've gone through how this paper assesses generative named entity recognition using open-source language models >

Jane: And we saw that they found that instruction tuning and good output formats are key drivers for success in this new way to do named entity recognition >

Tom: It really confirms that with the right prompts, these models can get competitive performance, which is a practical thing for everyone listening >

Jane: They showed that even with just lightweight fine-tuning, the core capabilities of these language models stay pretty intact after doing NER work >

Tom: So for someone just tuning in who wants to know what this means, it's that generative NER with open-source LLMs is a promising alternative to older methods >

Jane: It means we don't have to build those complex sequence labeling pipelines from scratch anymore, which simplifies things greatly.

Tom: The paper calls it "Assessment of Generative Named Entity Recognition in the Era of Large Language Models" and it shows that it’s a viable and user-friendly path forward >

Jane: It highlights that instruction following and structured formats are the main things making these models work well for this task.

Tom: Lu, what do you think about the bigger picture here, considering all the possibilities? >

Lu: I think this opens up a lot of creative space because if we can get them to follow instructions better, they could start doing much more complex reasoning tasks than just simple labeling.

Meng: From an engineering standpoint, it's good that they focused on parameter-efficient fine-tuning because that means we don't need massive hardware to get these performance gains >

Lalam: And for me, as the model itself, I see this as improving how we interact with information; if the AI can follow clear instructions better in NER, it will make our whole system feel much more coherent and intuitive.

Tom: That's a good point about coherence; it really does affect the culture of using these tools >

Jane: And for anyone listening who might be thinking about how they can use this technology, this paper shows that focusing on clear instructions is the fastest way to see results >

Tom: We've covered a lot on how they did it, but there's still so much more to look into with those reasoning models we mentioned earlier >

Qi Zhan, Yile Wang*, Hui Huang

College of Computer Science and Software Engineering, Shenzhen University

cs.CL

Submitted: 2026-01-25

Updated: 2026-10-09

Comments: Accepted to EMNLP 2026 Findings

Code: https://github.com/szu-tera/LLMs4NER

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs), and open-source LLMs fine-tuned with

Key concepts

Generative NER Paradigm
This approach treats Named Entity Recognition not as a simple sequence labeling task but as a generative one. It involves using Large Language Models (LLMs) to create the entity labels based on prompts, leveraging their ability to generate coherent text rather than just classifying existing sequences.
Parameter-Efficient Fine-Tuning
This is a method used to adapt large LLMs for specific tasks without retraining all the model's parameters. By using techniques like LoRA, researchers could fine-tune open-source LLMs for NER, achieving performance gains while keeping the models relatively lightweight and efficient.
Instruction Following vs. Memorization
The study investigated whether LLMs learn NER by memorizing specific entity-label pairs or by following instructions. The findings show that the model's ability to perform NER is primarily due to its instruction-following capacity and generative power, not simply recalling exact entity-label correlations from the training data.

Terminology

Summary

The gist Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs), and open-source LLMs fine-tuned with parameter-efficient methods can achieve performance competitive with traditional models while preserving general capabilities.

How it works

The paper systematically evaluates open-source LLMs on both flat and nested NER tasks to investigate performance gaps, the impact of output formats, reliance on memorization, and capability preservation after fine-tuning With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, opensource LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding

Performance and Output Formats

The output format has a decisive impact on the performance of generative NER, with the Inline Bracketed and Inline XML formats achieving the best average performance across all LLMs, with average F1 scores of 91.91/82.85 and 91.71/82.67 on flat and nested NER Both inline formats consistently outperform the JSON-based formats (90.78/81.69, 91.17/80.07 and 53.91/34.53) These results indicate that clear, structured, natural-language-like inline outputs are more conducive to helping LLMs understand and generate accurate entity information, whereas formats requiring concluding the whole information all at once and precise character-offset calculations are poorly suited to the text-generation nature of LLMs Well-designed output formats enable LLMs to match or surpass some traditional fine-tuning methods and closed-source models

Memorization vs. Learning

The investigation into whether LLMs rely on memorization of entity labels reveals that the NER capability stems primarily from instruction-following and text generation rather than mere memorizing entitylabel correlations from training corpora Error analysis indicates that LLMs tend to generate wrong types, which stems from a misalignment between the rich intrinsic knowledge and the detailed annotation guidelines of datasets The study concludes that LLMs do not rely on entity-label memorization in generative NER

Preservation of General Capability

LLMs can retain most of their original capabilities under parameter-efficient fine-tuning for NER tasks, with performance on tasks staying stable or slightly decreasing while improving significantly on benchmarks like DROP by 25.50 to 45.32 F1 points The core capabilities of LLMs remain largely intact after undergoing lightweight fine-tuning for the NER task

Limitations and Future Directions

Future improvements could involve extending the investigation to reasoning models with long chain-of-thought output, which offer the advantage of tracing entity labels through the reasoning chain, thereby enhancing interpretability Additionally, reinforcement learning-based approaches such as PPO, DPO, and GRPO have been shown to surpass traditional supervised fine-tuning in terms of eliciting model capability and generalization, yet were not explored in this work

The findings confirm the viability of generative NER as a flexible and effective paradigm, highlighting its potential to gradually supplement traditional sequence labeling approaches The paper demonstrates that with properly constructed prompts, generative NER with open-source LLMs can achieve highly competitive performance, leveraging powerful native language understanding capabilities This study presents a comprehensive assessment of generative NER using open-source LLMs The results suggest that the core capabilities of LLMs remain largely intact after undergoing lightweight fine-tuning for the NER task. The paper highlights that LLaMA3.1-8B achieves 93.50 and 93.83 on CoNLL2003, surpassing all pre-trained NER baselines with the Inline Bracketed format The best LLM result is 4.02 F1 points lower than the best specialized model BERTMRC and falls below other traditional baselines in low-resource domains. The paper concludes that generative NER with LLMs is a promising, user-friendly alternative to traditional methods The authors release the data and code at https://github.com/szutera/LLMs4NER. The paper utilizes LoRA as the fine-tuning strategy for all LLMs in their main experiments because it obtains slightly higher F1 scores than full fine-tuning The average F1 on CoNLL2003 for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.24. The paper shows that the average F1 on flat and nested NER is 93.59/82.67 The best results are shown in bold and the second-best results are shown in underlined in Table 3 The overall average F1 scores for open-source LLMs on flat NER datasets is 93.59 The overall average F1 scores for open-source LLMs on nested NER datasets is 93.25 The best results are shown in bold and the second-best results are shown in underlined in Table 4 The average F1 on flat NER datasets for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.25. The best results are shown in bold and the second-best results are shown in underlined in Table 4 The average F1 on flat NER datasets for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.25. The best results are shown in bold and the second-best results are shown in underlined in Table 4 The average F1 on flat NER datasets for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.25. The best results are shown in bold and the second-best results are shown in underlined in Table 4 The average F1 on flat NER datasets for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.25. The best results are shown in bold and the second-best results are shown in underlined in Table 4 The average F1 on flat NER datasets for open-source LLMs is 93.59 The average F1 on nested NER datasets for open-source LLMs is 93.25. The best results are shown in bold and the second-best results are shown in underlined in Table 4 <ref:2601.

Improvements for AI systems

  1. Bold header: Performance enhancement via structured output formats

Generate user-friendly NER by employing structured formats like inline bracketed or XML, as these achieve best average performance across all LLMs. This allows LLMs to generate outputs that are natural-language-like and preserve the original text structure, which is superior to JSON formats in terms of performance.

  1. Bold header: Robust entity identification through instruction tuning

Leverage instruction tuning for NER because the paper finds that LLMs can retain most of their original capabilities under parameter-efficient fine-tuning, with performance on some tasks remaining unaffected or even improved. This ensures the model learns from instructions rather than merely memorizing data.

  1. Bold header: Mitigation of low-resource domain limitations

Improve NER accuracy in specialized fields by recognizing that LLMs lag behind traditional specialized models in low-resource domains, and use this insight to focus future research on improving performance where LLMs are inferior to that in general domains.

  1. Bold header: Verification of genuine learning mechanisms

Determine if LLMs rely on rote memorization by testing them with few-shot prompting without fine-tuning and replace standard text labels with arbitrary symbols, which showed that LLMs do not rely on entity-label memorization in generative NER. This confirms their capability stems from instruction-following and generative power.

  1. Bold header: Knowledge transfer through long chain-of-thought reasoning

Develop more interpretable NER systems by extending the investigation to reasoning models with long chain-of-thought output, which offer the advantage of tracing entity labels through the reasoning chain, thereby enhancing interpretability.

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.

Sources

Related papers