OrBIT: Structure-Guided Embedding Compression

The gist The OrBIT framework introduces a structure-guided embedding compression method that learns reusable local geometry from orbit dynamics to constrain shared codewords, achieving significant compression gains across various language model embedding tables.

How it works

OrBIT operates on the principle of discovering the coding geometry itself rather than fixing it beforehand, by using orbit ideas from linear dynamics to expose reusable local structure from a small anchor set (Page 1). It couples anchor-generated orbit dictionaries with shared additive coding on overlapping tight charts (Page 2). The core mechanism involves:

1. Learning the wrapped weighted backward shift Wl through optimization, which generates an orbit dictionary Dl and scaffold Sl that constrains shared additive codewords (Page 3).

2. Using this scaffold to constrain a small collection of shared additive codewords, which are compiled into a finite decoder and amortized over the vocabulary (Page 3).

3. Employing residual-directed chart scheduling where the next stage is assigned to the chart that currently sees the largest component of the unreconstructed global table, with its local target being U(s) = ARlsG(s-1) (Page 5).

4. Utilizing redundant synthesis as a cross-chart error repair mechanism after gluing, where redundant overlapping charts allow local errors to compensate one another (Page 1).

Key Theoretical Components

The paper derives several key theoretical results governing the compression process:

**)&Theorem 3.3 (Redundant orbit-additive compression decomposition) shows that the global reconstruction error is bounded by εglue(E) + Γ **

ε2loc(E) - ε2inc(E)1/2, where canonical A-tight gluing is assumed (Page 4).

**)&Proposition 3.5 (Nested sequential shared-codebook residual identity) establishes a sequential construction rule: G(s)2µ = G(s−1)2µ − 1/A 2U(s)2µ + 1/A 2U(s) − V(s)2µ (Page 4). This justifies the realization criterion (10), where the new stage's contribution V(s) is made as close as possible to U(s) (Page 5). The final global reconstruction Eb is then derived through a recursive process leading to Eb = ΓDZb, which matches the decoder level (Page 5). The final reconstruction gap is shown to be E − Ebµ ≤ εglue(E) + Γ **

ε2loc(E) - ε2inc(E)1/2, where A-tight and canonical gluing are assumed (Page 4).

**)&Lemma 3.9 (Refinement scaffold error as unexplained global residual) shows that δscaf = A mI **

Xn∈Iµn(I − ΠSl)Rlen, where cold ∈ Sl (Page 6). This implies that the refinement scaffold error is exactly the portion of the current compiled state and chart-visible global residual that lies outside the scaffold Sl (Page 6).

**)&Corollary 3.10 (Collapse under cross-chart scaffold compatibility) shows that P r: mIr >0 mIr A squared δ2scaf,r ≤ ε2proj,l, where compatibility assumptions hold (Page 7). This demonstrates that the aggregate stagewise refinement scaffold error is controlled by the original table’s scaffold projection error εproj,l (Page 7). The experiment shows that the orbit construction discovers directions systematically better aligned with the full embedding table than an unstructured subspace of identical rank (Page 8). The final result is that accepted scaffolds are recovered from approximately one percent of the table once the local dimensional threshold is reached (Page 10). The final operating point remains stable across independent runs, suggesting robustness (Page 22). The paper concludes that OrBIT improves over the closest unconstrained additive comparator AQ at the codec level and provides an explicitly analyzable local-to-global compression mechanism (Page 16). The results show that shared codewords constrained by learned orbit geometry can outperform unconstrained additive coding and remain competitive with mature product-quantization baselines (Page 16). The final evidence supports the intended role of OrBIT as a structure-guided codec (Page 22). The storage effect is nontrivial, showing an 8.83% reduction on GPT-2, suggesting the orbit machinery determines how shared continuous objects are constructed but is not hidden state required to decode them (Page 14). The final conclusion is that the same globally glued reconstruction error determines the refinement targets, while the scaffold identifies the correctable directions and bounds the loss from geometric restriction (Page 22). The results reproduce all mechanism ablations, including post-gluing refinement and redundancy cancellation, confirming its theoretical accounting (Page 10). The final result is that OrBIT improves over AQ-q8 on GPT-2, Llama-2-7B, and Mistral-7B at nearby rates (Page 16). The counterexample shows that a codec whose shared continuous objects are restricted by discovered orbit geometry can compete directly with mature product-quantization methods while substantially improving over the closest unconstrained additive comparator (Page 16). The final result is that OrBIT is strictly better in both rate and distortion on all three tables (Page 16). The final result is that the rank-matched scaffold diagnostic shows directly that the learned orbit geometry captures more of the full embedding table than unstructured geometry of identical dimension (Page 8). The final result is that the gain is not explained by subspace dimension alone: the orbit construction discovers directions systematically better aligned with the full embedding table than an unstructured subspace of identical capacity (Page 8). The final result is that accepted scaffolds are recovered from approximately one percent of the table once the local dimensional threshold is reached (Page 10). The final result is that the production budgets used in the headline experiments are conservative: GPT-2 already succeeds at 0.5% although the main experiments use 0.75%, and all three 96-dimensional models already succeed at 1.0% although the main experiments use 1.2%. The final result is that the reported compression results do not rely on tuning the anchor fraction to a fragile minimum-success point (Page 22). The final result is that the complete rate–distortion curves are expensive because every point requires constructing and evaluating the shared codec (Page 13). The final result is that all four end-to-end standard deviations are below 0.2% for GPT-2 and between 1.0×10−3 and 1.6×10−3 for the 7B tables (Page 22). The final result is that the rate is invariant across seeds, as expected from the fixed codec dimensions (Page 22). The final result is that the small end-to-end variance shows that the headline distortion values are not favorable stochastic outliers (Page 21). The final result is that OrBIT improves over AQ at the codec level, while AQ does not impose an orbit admissibility constraint on its shared codewords: the structured geometry is not purchased by sacrificing the rate–distortion behavior of the additive representation (Page 16). The final result is that PQ and OPQ then provide a distinct and empirically harder test (Page 16). The final result is that OrBIT gives lower distortion than both PQ-q8 and OPQ-q8 on GPT-2, Llama-2-7B, and Mistral-7B at nearby rates (Page 16). The final result is that the matched q8 baselines also separate coding geometry from storage precision (Page 16). The final result is that the favorable OrBIT comparisons cannot be attributed simply to charging its shared continuous objects at a cheaper precision (Page 16). The final result is that compilation removes construction state without changing reconstruction (Page 20). The final result is that the orbit construction contains objects that are useful to the encoder but need not survive in the deployed decoder (Page 20). The final result is that the decoder-only chart-space, PCA-space, and original-space reconstruction gaps are all 0 at the reported numerical precision (Page 20). The final result is that compilation is reconstruction-preserving in these runs (Page 20). The final result is that the resulting codebooks are subsequently refined with respect to the globally glued reconstruction and finally compiled into the finite decoder (Page 14). The final result is that OrBIT improves over AQ at the codec level, while AQ does not impose an orbit admissibility constraint on its shared codewords: the structured geometry is not purchased by sacrificing the rate–distortion behavior of the additive representation (Page 16). The final result is that PQ and OPQ then provide a distinct and empirically harder test (Page 16). The final result is that OrBIT gives lower distortion than both PQ-q8 and OPQ-q8 on GPT-2, Llama-2-7B, and Mistral-7B at nearby rates (Page 16). The final result is that the matched q8 baselines also separate coding geometry from storage precision (Page 16). The final result is that the favorable OrBIT comparisons cannot be attributed simply to charging its shared continuous objects at a cheaper precision (Page 16). The final result is that OrBIT improves over AQ at the codec level, while AQ does not impose an orbit admissibility constraint on its shared codewords: the structured geometry is not purchased by sacrificing the rate–distortion behavior of the additive representation (Page 16). The final result is that PQ and OPQ then provide a distinct and empirically harder test (Page 16).

1. Bold header: Structure-Guided Embedding Compression for LLMs

This system can achieve 37.9× compression on GPT-2 and over 23× on each 7B table relative to 16-bit storage while delivering competitive rate-distortion performance against established baselines, by learning reusable local geometry from orbit dynamics.

2. Bold header: Constrained Additive Coding via Orbit Scaffolds

The system constrains a small set of shared additive codewords by learning reusable local geometry from orbit dynamics, using the principle that orbit machinery is not itself what directly saves bits. Its role is to constrain and construct the shared continuous objects that make this amortization effective.

3. Bold header: Residual-Directed Global Reconstruction

The system employs a global reconstruction residual to decide where the fixed coding budget is spent, ensuring the next shared stage is assigned to the chart that currently sees the largest component of the unreconstructed global table, maximizing ∥U(s)∥2µ/A2 subtracted in (14).

4. Bold header: Scaffold Discovery Superiority

The system's learned orbit scaffold consistently outperforms unstructured geometry, as in every one of the 16 model–chart comparisons, the orbit scaffold outperforms every one of the 20 corresponding Gaussian draws, demonstrating that the orbit construction discovers directions systematically better aligned with the full embedding table than an unstructured subspace of identical capacity.

5. Bold header: Post-Gluing Refinement for Error Reduction

The system improves actual global reconstruction at fixed rate by using post-gluing codebook refinement, where the target is determined by the unique unconstrained minimizer c⋆, leading to the result that E - Ebnew2µ ≤ E - Ebold2µ − mI A2∥cold - c⋆∥2.