The GELATO Dataset for Legislative NER

arXiv:2603.14130 · cs.CL · Submitted 2026-03-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The GELATO Dataset for Legislative NER".

Jane: Here's the gist of the paper. The gist The GELATO dataset introduces a two-level named entity recognition ontology for U.S.

Tom: congressional legislation, which is important for supporting various types of research and more advanced NLP methods and applications How it works The paper introduces GELATO, a two-level NER ontology designed for U.S.

Jane: congressional legislation This ontology consists of six top-level classes and 30 subclasses tailored to U.S.

Tom: legislative text The top level has six tags: Person, Organization, Document, Act, Abstraction, and Class The two levels of labels for each document and mention in GELATO were completed in two steps with three total annotators The first level of GELATO annotation for each document was completed by two annotators using Label Studio All three annotators participated in the adjudication process, and all tokens with differing tags were discussed by the group of annotators and resolved through group consensus To obtain the second-level GELATO labels, the three annotators reviewed the gold labels together For each gold label mention, the annotators decided which second-level label of the respective first-level label was most appropriate, and tagged it accordingly Dataset Details The dataset consists of 131 U.S.

Paper summary: Tom: So to recap this segment, we're talking about "The GELATO Dataset for Legislative NER". The core idea is that they introduced a dataset of U.S. House and Senate bills from the 118th Congress annotated with a new two-level named entity recognition ontology designed for U <ref:2603.14130#pg1,House and Senate bills from the 118th Congress annotated>.S. legislative texts >

Jane: They claim this matters because it supports various types of research and more advanced NLP methods, specifically for legislative NER or other downstream tasks where you need to extract specific entities from congressional documents >

Tom: That’s right. The thesis is that by fine-tuning models like BERT and RoBERTa on this GELATO dataset for the first-level prediction, and then using LLMs with optimized prompts to finish the second level prediction, we can get a good look at how these combinations work >

Lu: They are essentially testing if this two-stage approach is effective for legislative text entities >

Jane: It’s about seeing how strong RoBERTa is for the first step and where the LLMs step in for that more complex second level labeling >

Tom: The results show that RoBERTa has a strong F1 score of eighty-eight point seven two on the first level, while BERT models are relatively weaker > <ref:2603.14130#pg3>

Meng: That’s a specific number to look at. So, what does that eighty-eight point seven two tell us about the quality of the initial entity extraction <ref:2603.14130#pg3>?

Tom: It tells us that RoBERTa is performing very well for this task when it comes to identifying the first set of entities in legislative text >

Jane: And then they use LLMs to handle the more granular second level predictions, and while BERT performs worse at that stage, the LLM can still complete the task >

Lu: That supports future research into using these model combinations as extraction tools for legislative NER because it shows a potential path forward >

Meng: But we have to be careful about those LLMs, right? They bring their general knowledge with them, which means their performance isn't guaranteed on this specialized data >

Jane: Exactly. The paper also points out that the similarity between entities in the Person and Class classes is subtle, and future work needs to explore why models might be missing those class predictions >

Tom: So the main takeaway here is that GELATO provides a structured way to benchmark model performance for legislative NER using this specific two-stage pipeline >

Lu: And it’s a useful benchmark for investigating the performance of models at NER in the legislative domain because it sets a standard for what to expect when working with this specific type of text >

Tom: The structure of GELATO is what makes it a useful benchmark for seeing how different model architectures handle the task >

Conclusion: Jane: We’re wrapping up our discussion on "The GELATO Dataset for Legislative NER". The authors are Matthew Flynn, Timothy Obiso, and Sam Newman from Brandeis University >

Tom: So the key implication is that this paper provides a structured benchmark to investigate model performance at Named Entity Recognition specifically in the legislative domain >

Jane: In simple terms, it shows us how important it is to tailor our data and our models when we are working with specialized text like congressional bills rather than just general news articles >

Tom: It also suggests that a two-stage prediction pipeline, using a strong model for the first level and an LLM for the second level, is a viable way to approach complex extraction problems in this area >

Lu: The finding that RoBERTa did better on the first level and BERT struggled there tells us we need to pay attention to which base models we choose when building our pipelines >

Meng: And we have to consider that the limitations they mentioned, like the dataset being limited to bills one hundred through two hundred of the 118th Congress > <ref:2603.14130#pg2>

Jane: Which means for practical application, we shouldn't assume this performance holds across all legislative texts without further testing >

Tom: So in a nutshell, GELATO is a useful resource for benchmarking how different AI models handle the specific challenges of legislative text NER and it points toward more targeted research down that road >

Lu: It’s a structured approach to annotation that sets a good precedent for creating high-quality datasets in specialized domains >

Meng: And we still need to ensure that whatever we build based on this data can handle the scale and the complexity of real legislative documents without breaking at scale >

Matthew Flynn, Timothy Obiso, Sam Newman

Brandeis University

cs.CL

Submitted: 2026-03-14

Updated: 2026-03-14

Comments: Accepted at LREC 2026

DOI: 10.63317/3axxkz9oh5th

Code: https://github.com/Wollaston/gelato

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The GELATO dataset introduces a two-level named entity recognition ontology for U.S.

Key concepts

GELATO Ontology: This is a specialized, two-level tagging system created specifically for U.S. legislative texts. It has six main top-level categories (like Person or Act) and thirty specific subcategories tailored to how entities appear in congressional bills, making it precise for legislative research.
Two-Level NER: This is a two-step process used to label text entities. First, models predict the broad category (Level 1), and then a separate method, like an LLM prompt, refines that prediction into a more specific category (Level 2). This helps capture complex relationships within legal documents.
Transformer Models: These are advanced deep learning architectures (like BERT and RoBERTa) used to analyze text. They are fine-tuned on the GELATO data to accurately predict the first level of entity tags, demonstrating their capability in understanding legislative language.
LLMs as Second-Level Predictors: Large Language Models (like Qwen) are used after the initial model prediction to make more detailed, granular classifications. This combination tests whether general knowledge from LLMs can effectively refine the specific predictions made by smaller transformer models.

Terminology

Ontology — 2× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The GELATO dataset introduces a two-level named entity recognition ontology for U.S. congressional legislation, which is important for supporting various types of research and more advanced NLP methods and applications

How it works

The paper introduces GELATO, a two-level NER ontology designed for U.S. congressional legislation This ontology consists of six top-level classes and 30 subclasses tailored to U.S. legislative text The top level has six tags: Person, Organization, Document, Act, Abstraction, and Class

The two levels of labels for each document and mention in GELATO were completed in two steps with three total annotators The first level of GELATO annotation for each document was completed by two annotators using Label Studio All three annotators participated in the adjudication process, and all tokens with differing tags were discussed by the group of annotators and resolved through group consensus To obtain the second-level GELATO labels, the three annotators reviewed the gold labels together For each gold label mention, the annotators decided which second-level label of the respective first-level label was most appropriate, and tagged it accordingly

Dataset Details

The dataset consists of 131 U.S. House and Senate bills from the 118th Congress annotated using the GELATO tag set All data was queried from the Congress.gov API The training (n = 80) and dev (n = 21) sets are made up of House bills; and the test set contains only Senate bills (n = 30) The training set consists of House bills with a total of 77,372 tokens The dev set also consists of House bills with a total of 23,819 tokens The test set contains Senate bills with a total of 31,740 tokens Table 5 summarizes the best F1 scores and hyperparameter configuration for each model

Model Performance

The paper fine-tunes transformer-based models (BERT, RoBERTa) on the GELATO dataset for first-level predictions The results show the strengths and weaknesses of models of different sizes and pretraining strategies The best-performing model is roberta-large with an F1 of 88.72 For the level-two prediction task, in which the Qwen model makes fine-grained predictions based off of level-one labels, the F1 score is approximately 20 points lower for RoBERTa models, and approximately 14 to 15 points lower for BERT models

Error Analysis and Discussion

BERT models performed much worse than RoBERTa models The superior performances of the RoBERTa model emphasizes how a larger vocabulary size and more diverse pretraining data can greatly improve model performance on the same task for a model of a similar architecture The complete absence of Class in most model predictions indicates an open challenge in legislative NER The similarity between items in the Person and Class classes are subtle; future work can explore why and how models may need more training The models in our experiments also differ in their vocabulary sizes and tokenizers As for LLMs, they have vast amounts of general knowledge from their pretraining, and this can impact performance when these general expectations do not align with those of GELATO This suggests that when designing NER ontologies that LLMs will use, it is important to have clear semantic distinction to avoid LLM confusion

Limitations and Ethical Considerations

The GELATO dataset is limited to House and Senate Bills 100-200 from the 118th Congress, which limits our dataset temporally and stylistically Treaty and Case have no mentions across our train, dev, and test sets which limits our ability to use this dataset to extract those entity types One limitation of our two-stage prediction pipeline is that errors made in stage one propagate through the pipeline and are irrecoverable in stage two Automated entity extraction could also enable potentially harmful uses such as targeted analysis of how specific groups are referenced in legislation or identification of individual legislators for inappropriate purposes

The paper releases its code and models under a permissive open-source license to encourage more research with our models and data The authors created GELATO through a two-stage process with full adjudication of any disagreements The GELATO can support beneficial applications including legislative tracking, policy analysis, and government transparency initiatives

The paper is a useful benchmark for investigating the performance of models at NER in the legislative domain The GELATO dataset is limited to House and Senate Bills 100-200 from the 118th Congress

The paper is a useful benchmark for investigating the performance of models at NER in the legislative domain<ref:2603.

Improvements for AI systems

Improved entity extraction pipeline: Implement a two-stage NER system where transformer models on GELATO data perform level one prediction, followed by LLMs with optimized prompts to complete second-level prediction, as this supports future research in legislative NER or downstream tasks using these model combinations as extraction tools.
Enhanced model selection for first-level prediction: Utilize RoBERTa models over BERT models for first-level classification due to superior performance, specifically noting that the best-performing model is roberta-large with an F1 of 88.72 compared to BERT base (F1 of 68.09).
Fine-grained second-level prediction: Employ a no-training approach for level two NER label prediction using LLMs optimized via DSPy, allowing the system to take advantage of the strengths of LLMs and smaller, fine-tuned transformer language models.
Specific entity type identification: The system can accurately distinguish between Person and Class mentions by leveraging the ontology's distinction, addressing confusion where LLMs frequently confused the Name subclass of the firstlevel Person class with the other Member subclass.
Robust handling of cascading errors: Since any error made in stage one would cascade to stage two, the system will prioritize high-performing level one predictions, as this is expected that level-two predictions generated from RoBERTa predictions would strongly outperform those from BERT models.
Optimized prompt engineering for LLMs: Develop six optimized prompts, one for each level-one label, using a shared system prompt structure to ensure the LLM adheres to the required output format when predicting second-level labels.

Abstract

This paper introduces GELATO (Government, Executive, Legislative, and Treaty Ontology), a dataset of U.S. House and Senate bills from the 118th Congress annotated using a novel two-level named entity recognition ontology designed for U.S. legislative texts. We fine-tune transformer-based models (BERT, RoBERTa) of different architectures and sizes on this dataset for first-level prediction. We then use LLMs with optimized prompts to complete the second level prediction. The strong performance of RoBERTa and relatively weak performance of BERT models, as well as the application of LLMs as second-level predictors, support future research in legislative NER or downstream tasks using these model combinations as extraction tools.