DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods

The gist The results show that automated extraction is feasible at a high level of accuracy for many commonly studied provisions, with median performance near the upper bound across approaches.

Introduction and Problem

Much empirical legal research depends on translating unstructured text into structured variables, which traditionally has relied on human coders who read and code legal documents such as corporate charters and bylaws<ref:2607.15879#pg2>. Legal texts present distinctive challenges for natural language processing because they are often lengthy, structurally complex, and written in specialized language that varies across documents<ref:2607.15879#pg3>. The paper introduces DECODEM, a set of benchmark datasets for evaluating the automated extraction of corporate governance variables from organizational documents<ref:2607.15879#pg4>. These benchmarks pair randomly sampled corporate charters and bylaws with high-quality human annotations covering provisions commonly studied in empirical work<ref:2607.15879#pg4>.

Methodology and Benchmarks

The underlying task consists of a set of document-level binary classification problems, one for each governance variable<ref:2607.15879#pg4>. The DECODEM benchmarks are constructed from a corpus of real-world corporate governance documents and focus on provisions governing shareholder meetings, director elections, board structure, indemnification, exculpation, and takeover-related arrangements<ref:2607.15879#pg4>. For each provision, the datasets provide hand-coded labels that capture whether the relevant governance arrangement is present<ref:2607.15879#pg4>. The charters benchmark contains 300 charters, while the bylaws benchmark contains 150 bylaws<ref:2607.15879#pg6>. All variables take the form of binary indicators, which facilitates model evaluation<ref:2607.15879#pg6>.

Extraction Pipelines and Performance

The paper evaluates several extraction pipelines that vary along dimensions including the complexity of the prompts used to query the model and the extent to which the model is provided with the full document or with selected excerpts identified through a pre-processing step<ref:2607.15879#pg5>. The results show that large language models can recover many governance variables with high accuracy<ref:2607.15879#pg5>. Performance varies substantially across variables, with a small number of arguably more complex provisions—such as shareholder meeting rights and removal and vacancy-filling authority—accounting for most of the observed errors<ref:2607.15879#pg5>. More elaborate prompting strategies and cascading extraction pipelines do not consistently improve performance for frontier models, but they substantially narrow the gap between frontier and efficiency-oriented models in some settings<ref:2607.15879#pg5>.

Comparative Analysis of Architectures

The study compares five extraction approaches across three dimensions: the scope of extraction (global vs. topic-specific), the level of prompt detail, and whether the model operates on full documents or selected excerpts<ref:2607.15879#pg8>. The Global approach provides the full bylaw to extract all governance variables in a single pass, while Detailed Cascade provides detailed prompts and relies on identifying relevant parts of a bylaw before feeding the text into the model<ref:2607.15879#pg8>. For frontier models, Global and Detailed Cascade yield broadly similar overall performance<ref:2607.15879#pg9>. However, Detailed Cascade substantially improves performance for the efficiency-oriented models, with gains distinguishable from zero in all three model families<ref:2607.15879#pg9>.

Document Type Comparison

The evaluation extends to the charters benchmark, testing whether the high-accuracy findings reported for bylaws generalize to a second class of organizational document<ref:2607.15879#pg8>. While average performance on the charters benchmark trails that on the bylaws benchmark, most shared variables cluster in the upper-right corner, indicating that extraction performance is broadly stable across document types<ref:2607.15879#pg14>. A few variables depart from this pattern, such as EXC-O, which lies well above the line, reflecting markedly better performance on charters than on bylaws<ref:2607.15879#pg15>.

Conclusion and Implications

The results reinforce the high-level picture that most governance provisions can be identified with high accuracy using relatively simple prompting strategies<ref:2607.15879#pg14>. The main obstacle to further gains for the strongest models lies less in extraction architecture than in the interpretive difficulty of particular provisions<ref:2607.15879#pg10>. These findings demonstrate that large language models can substantially reduce the cost of constructing governance datasets, potentially enabling empirical research at a scale that was previously infeasible<ref:2607.15879#pg10>. The paper highlights the importance of careful task design and validation, as certain provisions remain difficult to extract and require particular attention in downstream applications<ref:2607.15879#pg10>.

1. Develop extraction pipelines that incorporate both prompt detail and document handling choices to maximize performance across models. This allows AI systems to leverage cascading pipelines or full-document approaches depending on the specific governance variable being extracted, as results show that "more elaborate prompting strategies and cascading extraction pipelines do not consistently improve performance for frontier models, but they substantially narrow the gap between frontier and efficiency-oriented models in some settings."

2. Implement model-specific tuning strategies based on variable complexity. AI systems can be designed to use simpler prompts for Simple Variables while employing detailed prompts for Complex Variables, as performance varies systematically across variables, with errors concentrated in those requiring more complex interpretation.

3. Create adaptive extraction architectures that utilize pre-processing steps to reduce token count without sacrificing recall. This allows AI systems to employ excerpting pipelines generated by a BERT- and LASSO-based pipeline that predicts, for each paragraph, whether it contains language relevant to a given governance provision, thereby reducing extraction costs while preserving the information needed to identify relevant provisions.

4. Design systems capable of identifying and mitigating model biases related to specific document types. Since performance on charters trails bylaws for certain variables due to factors like provisions applicable exclusively to holders of specific series of preferred stock, AI systems should be trained with knowledge that these structural features might cause borderline coding decisions or coder mistakes rather than a genuine extraction failure.

5. Utilize post-extraction adjudication processes to refine variable labels based on model disagreements. The system can employ a process where every observation on which the initial human coding, the benchmark labels, and six of the extraction routines described below do not unanimously agree is recoded blindly, which helps in creating Adjudicated Benchmarks to improve accuracy by addressing discrepancies.

Much empirical legal research depends on translating unstructured text into structured variables. In corporate governance research as elsewhere, this translation has traditionally been performed by humans. Manual coding is time-consuming and costly. In corporate governance research, standardized commercial datasets offer an alternative, but they cover a limited set of firms and governance variables. This paper introduces the DECODEM Benchmarks and uses them to evaluate whether large language models can extract corporate governance variables from charters and bylaws. The benchmarks combine a sample of corporate charters and bylaws of publicly traded U.S. corporations obtained from the SEC's EDGAR database with hand-coded labels that capture whether a document includes a specific governance arrangement. The paper uses these benchmarks to evaluate a range of extraction approaches and models. The approaches vary along several dimensions, including the amount of guidance included in the prompts and the extent to which the model is provided with the full document or with relevant excerpts only. The results show that current models can recover many governance variables with high accuracy across extraction approaches. At the same time, performance varies substantially across documents and variables. In particular, extraction is less accurate for charters than for bylaws, including when the comparison is limited to variables that appear in both benchmarks. The results also show that frontier models perform comparably on the tasks evaluated here and that more elaborate extraction pipelines do not consistently improve their performance. More elaborate extraction strategies matter more for efficiency-oriented models, where they can substantially narrow the performance gap relative to frontier models.