FRAGMENT: Factorized Graph Representations for Document Generation and Editing via Entity-Aware Transformations

The gist The framework explores representing documents as typed relational graphs and factorizes their distribution as p(structure,content) = p(structure)· p(content structure).

FRAGMENT Overview

FRAGMENT is a generative framework that represents a document as a typed relational graph and factorizes its distribution as p(structure,content) = p(structure)·p(content structure) The framework consists of two stages: the first stage, the Architect, is a causally masked Transformer conditioned on the document category that autoregressively generates the document graph The second stage, the Builder, is a GATv2-based graph attention network with residual pre-normalized message passing that enriches the generated graph with normalized bounding boxes, textual content, and visual attributes through dedicated prediction heads. Both stages define explicit likelihood models. This factorized formulation yields a tractable document-level likelihood that serves as an anomaly score for synthetic and manipulated document detection.

Document Representation and Factorization

Documents are represented as typed attributed graphs G = (V,E, τ,ρ,A) where each node v ∈ V corresponds to a semantic layout element with type τ(v) ∈ T (e.g., title, text, table), each directed edge e ∈ E carries a relation type ρ(e) ∈ [BELOW, RIGHT-OF], and A collects node attributes: a normalized bounding box bv in (x1,y1,x2,y2) form, a token sequence tv (WordPiece, max length 50), and style attributes (sv, fv,cv) for font size (normalized to [0,1]), font family (categorical over a normalized font vocabulary), and RGBA color. The first term in the factorization corresponds to the generation of the document scaffold, including semantic entities and their relations, while the second term models the realization of this scaffold through geometry, text, and visual attributes.

The Architect Stage

The Architect models p(structure) = p(V,E, τ,ρ d) autoregressively A document graph is serialized as a sequence of construction steps; step i emits (i) the type of node vi, (ii) a pointer to the existing node that vi attaches to, and (iii) the type of the resulting edge. The backbone is a Transformer encoder applied with a causal mask Category conditioning is implemented by prefixing, where a learned embedding of the document category d is prepended to the sequence. Three heads read the hidden states: a node-type classifier, an edge-type classifier, and a pointer mechanism that scores attachment targets by a scaled dot product between a query projection of the current state and key projections of all previous states.

The Builder Stage

The Builder models p(A G,d) = ∏v∈V p(bv hv)p(tv hv)× p(sv hv)p(fv hv)p(cv hv), where hv denotes the contextual representation of node v produced by the GATv2 message-passing backbone. The message-passing backbone uses four GATv2 layers with 8 heads, edge-type embeddings as edge features, pre-layer-normalization, and residual connections. Each node state feeds three prediction heads: Layout (an MLP with sigmoid output predicting bˆv), Style (separate sub-heads for font size, font family, and RGBA color), and Text (a 2-layer GRU decoder). The text head is modular, allowing the GRU to be replaced by a Transformer decoder or an instruction-tuned language model.

Editing and Detection Capabilities

The framework supports two auxiliary capabilities: prompt-conditioned editing via cross-attention and the composite log-likelihood used for forgery detection. For editing, the instruction is encoded by a frozen pre-trained BERT encoder into a sequence of embeddings C Inside the Builder, each message-passing block is augmented with a multi-head cross-attention sublayer in which node states query the instruction. An edit mask, derived by thresholding the cross-attention mass each node receives, restricts re-generation to affected nodes. For forgery detection, the composite log-likelihood is defined as log p(G,A d) = log p(G d) + log p(A G,d). This decomposition across nodes and attribute heads enables structural and attribute-level anomaly localization.

Evaluation and Findings

Experiments on DocLayNet, FUNSD, and SROIE evaluate FRAGMENT alongside representative autoregressive, layout-only, and graph-based baselines. The results illustrate that FRAGMENT exhibits competitive Doc-EMD scores on layout metrics, and the decomposable likelihood provides an informative score without requiring explicit tampered training data. Ablation results highlight component trade-offs, such as substituting GATv2 with GCN leading to lower layout and style scores. Future extensions include incorporating higher-capacity text decoders and exploring continuous generative models for attribute generation. The framework serves as a useful exploration into structured, controllable, and verifiable document generation.

How it works

(a) Architect scaffold

The Architect autoregressively generates a typed document graph conditioned on a category prefix This stage models p(structure) = p(V,E, τ,ρ d).

(b) Builder realization

The Builder instantiates this scaffold by predicting geometric, textual, and visual attributes It models p(A G,d) = ∏v∈V p(bv hv)p(tv hv)× p(sv hv)p(fv hv)p(cv hv).

(c) Rendered page

The final output is the rendered document, which incorporates layout attributes, textual content, and visual styling.

Summary of Contributions

1. We formulate a unified graph representation for structured documents together with an R-tree-based construction procedure that extracts typed spatial relations from layout annotations.

2. We propose a two-stage generative architecture that separates structural generation from content realization through an autoregressive graph-based Architect and a GATv2-based Builder.

3. We define a composite document log-likelihood that provides a measure of document plausibility for forgery detection.

4. We introduce a prompt-conditioned editing mechanism based on cross-attention that supports semantic and entity-aware document modification.

5. We present the training methodology and an evaluation protocol examining layout fidelity, semantic quality, and forgery detection across benchmark datasets.

Limitations

The two spatial edge types capture Manhattan-style layouts effectively but provide limited representation for nested or deeply hierarchical structures. Additionally, the GRU text decoder presents capacity limitations for long text fields, and OCR errors can propagate during end-to-end editing workflows. Finally, likelihood-based anomaly detection requires per-category calibration to maintain score consistency across different document domains<ref:2608.

1. No longer rely solely on pixel or token-level generation by implementing FRAGMENT to represent documents as a typed relational graph and factorizing its distribution as p(structure,content) = p(structure)· p(content structure). This allows for explicit modeling of document structure while preserving the flexibility required for content generation.

2. Develop an AI system capable of anomaly scoring by calculating the composite document log-likelihood defined by Equation 5: log p(G,A d) = log p(G d) Architect, Eq. 2 + log p(A G,d) Builder, Eq. 4. This provides a measure of document plausibility for forgery detection that is localized to structural and attribute-level anomaly localization.

3. Implement controlled document editing by utilizing the prompt-conditioned extension which inject[s] instruction embeddings into the Builder through cross-attention, enabling semantic and entity-aware modifications. This allows for operations like updating fields based on instructions such as “add a 15% tip” while preserving untouched content.

4. Enhance document understanding by leveraging the explicit intermediate representations provided by FRAGMENT, which separates generation into two complementary stages where the Architect first generates a typed relational scaffold describing the semantic entities and their relations. This allows for inspecting and constraining the structural plan before content realization.