SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

The gist The proposed Static Orthogonal Subspace LoRA (SOS-LoRA) reparameterizes a rank-rtot update as a sum of K static low-rank experts with fixed multi-scale scaling and explicit cross-expert diversification of input-side directions, while preserving LoRA’s mergeability and inference efficiency.

Problem Identification

Standard Low-Rank Adaptation (LoRA) assigns each adapted matrix a single low-rank update pathway, which raises the natural question: under a fixed rank budget, can one shared low-dimensional pathway adequately support the diverse behaviors required by modern LLMs? In the standard factorization, the adaptation signal depends on the input only through a single projection XA, i.e., a single set of input-side directions shared by all adaptation effects.> Consequently, heterogeneous behaviors may be forced to share and compete for the same inputside directions under a fixed rank budget, which can induce interference and optimization coupling.> Increasing the rank can partially alleviate this issue, but it also increases trainable parameters and still provides no explicit mechanism for structured adaptation under a fixed budget.>

SOS-LoRA Architecture Design

SOS-LoRA mitigates this coupling by structuring the low-rank update into multiple expert components without increasing the total rank budget.> We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in replacement for standard LoRA that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts.> Each expert uses its own inputside directions, is assigned a fixed scale, and is explicitly encouraged to remain directionally distinct from other experts.> SOS-LoRA instantiates this inductive bias through three components:

1. pert decomposition splits the total rank budget into K experts of rank r′ = rtot/K, matching the parameter budget of standard LoRA at the same rtot.>

2. a fixed multi-scale scaling scheme assigns each expert a static scale spanning small-to-large magnitudes; this scales the raw backpropagated gradients entering each expert and encourages scaleseparated learning dynamics.>

3. orthogonal input-side diversification combines cross-expert orthogonal initialization (with zero initial update) and a lightweight regularizer that penalizes crossexpert correlations in input-side directions.>

The SOSLoRA forward pass is defined by YSOS = XW0 + X Kk1 sk XAkBk, where sk > 0 controls the contribution of expert k.>

Key Mechanisms

The design incorporates several mechanisms to ensure structured adaptation under a fixed budget:

((

Fixed multi-scale base scaling. We initialize expert scales as sk = α/rtot · γk / γbar, so that the average scaling matches standard LoRA, 1/K PKk1 sk = α/rtot, while experts receive linearly spaced relative scales γk ∈ [1, γmax]. This results in scale-separated gradient signals.>

(The per-expert gradients satisfy ∂L/∂Bk = sk (XAk)⊤∆, ∂L/∂Ak = sk X⊤(∆B⊤k), so sk linearly scales the raw backpropagated gradients entering expert k.)

Cross-expert input-side diversification is achieved through:

1. Cross-expert orthogonal initialization with zero initial update. Assuming rtot = Kr′ ≤ m, we initialize experts to span diverse input-side directions while keeping the initial update exactly zero.>

2. Cross-expert input-direction regularization. We regularize the input-side directions by penalizing cross-expert correlations between the column spaces of Ak.> This is defined by Lorth = λ r′2 X1≤k<l≤K A˜⊤k A˜l 2 F, which equals the sum of squared cosine similarities between all pairs of normalized columns across different experts.>

Empirical Results and Analysis

Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants.> On Llama 2-7B, SOS-LoRA reaches 85.5 on average, improving over PiSSA (84.5) by 1.0 point and standard LoRA (77.6) by 7.9 points.> SOS-LoRA also achieves the best result on all eight benchmarks in this setting, including PIQA (87.5 vs. 87.0 for PiSSA), and attains the best overall average.> In encoder-based NLU settings, SOS-LoRA achieves the highest average score (88.3) among the compared PEFT methods, improving over the strongest PEFT baselines under our evaluation pipeline (Delta-LoRA/MELoRA, 87.5) by 0.8 points.>

Cost and Efficiency

SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging.> The merged update is ∆W = PKk1 skAkBk, which can be absorbed into W0 with no additional inference-time parameters or latency after merging.> Relative to LoRA, SOS-LoRA improves by 14.35 points on GSM8K and 4.14 points on MATH.> This gain comes with a modest increase in training cost (4:52 vs. 4:31 wall-clock time; 67.45 vs. 66.32 GB peak reserved VRAM), likely due primarily to the orthogonal regularizer.> Overall, SOS-LoRA offers a favorable accuracy–cost trade-off under this protocol.>

Conclusion

SOS-LoRA identifies shared input-side coupling as a limitation of low-rank adaptation under a fixed rank budget, where a single input-side pathway can couple heterogeneous adaptation signals and induce optimization interference.> We propose Static Orthogonal Subspace LoRA (SOS-LoRA), which decomposes each low-rank update into K static (always-on, non-routed) experts with a fixed multiscale scaling scheme and explicit input-side direction diversification via orthogonal initialization and a lightweight regularizer.> SOS-LoRA is mergeable into the base model, adding no inference-time parameters or latency after merging.> Experiments on both encoder-only (RoBERTa) and decoder-only (Llama 2/3) backbones show consistent gains over PEFT baselines under matched training budgets, suggesting that explicit input-side direction diversification can provide a practical route to stronger parameter-efficient adaptation.>

Limitations

Our evaluation covers both encoder-only (RoBERTa-base) and decoder-only (Llama 2/3) backbones, but it is still limited to a relatively narrow range of model sizes and benchmark suites.> Performance in larger models and broader settings (e.g., long-context, multilingual, and tool-use scenarios) remains untested.> SOS-LoRA also introduces additional hyperparameters (e.g., expert count K, multi-scale range γmax, and regularization weight λ). Although we use a consistent default configuration and provide ablations and sensitivity analyses for key factors, the best settings may still vary with architecture, task, and training recipe.> Finally, our analysis relies on geometric proxies (e.g., cross-expert squared-cosine similarity between input-side directions and relative update magnitudes) that capture redundancy and scale separation, but do not directly measure functional specialization.>

Acknowledgments

This work is supported by the National Key Research and Development Program of China (No.2023YFF0905400), the National Natural Science Foundation of China (No.U2341229) and the Reform Commission Foundation of Jilin Province (No.2024C003).

References

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255.>

AI@Meta. 2023. Llama 2: Open foundation and finetuned chat models. CoRR, abs/2307.09288.>

AI@Meta. 2024. Llama 3 model card.>

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language.>

Yupeng Chang, Chenlu Guo, Yi Chang, and Yuan Wu. 2026. BAloRA: Bias-alleviating low-rank adaptation to mitigate catastrophic inheritance in large language models.>

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang>, and 1 others. 2024. A survey on evaluation of large language models.>

Christopher Clark, Kenton Lee, Ming-Wei Chang et al. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions.>

Peter Clark, Isaac Cowhey, Oren Etzioni et al. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.>

Karl Cobbe, Vineet Kosaraju et al. 2021. Training verifiers to solve math word problems.>

Haonan Dong, Kehan Jiang, Haoran Ye et al. 2026. Neureasoner: Towards explainable, controllable, and unified reasoning via mixture-of-neurons.

1. Bold header: Static Orthogonal Subspace LoRA for Heterogeneous Adaptation

SOS-LoRA reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts, which allows the system to support diverse behaviors by encouraging diverse input-side directions of adaptation while maintaining a fixed rank budget.

2. Bold header: Improved Reasoning and Knowledge Benchmarks

The system can achieve consistent gains over matched-budget LoRA baselines and recent variants on reasoning and knowledge-intensive benchmarks like GSM8K/MATH, as evidenced by SOS-LoRA reaching 97.2 accuracy on PIQA under the proposed setup.

3. Bold header: Enhanced Encoder-Based Natural Language Understanding

The model demonstrates superior performance on NLU tasks, achieving a score of 88.3 on GLUE with RoBERTa-base, surpassing full fine-tuning in this setting by 0.8 points.

4. Bold header: Optimized Cross-Expert Separation

By employing an orthogonality regularizer, the system actively discourages redundancy by penalizing cross-expert correlations in input-side directions, leading to a lower off-diagonal similarity matrix compared to non-regularized variants.

5. Bold header: Stable Optimization Dynamics

The fixed multi-scale scaling scheme ensures that scaleseparated learning dynamics are promoted, resulting in faster early loss reduction and lower final loss during training compared to uniform scaling controls.

6. Bold header: Deployment Efficiency Preservation

SOS-LoRA preserves LoRA’s key deployment property because the merged update is absorbable into the pretrained weights, adding no inference-time parameters or latency after merging.