ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

The gist: ZOMP proposes a query-efficient, fully forward-only method that tunes deep prompts in both vision and text branches of frozen CLIP models using simultaneous perturbation stochastic approximation to achieve state-of-the-art performance under realistic query budgets.

How it works

ZOMP is designed to address the infeasibility of backpropagation in prompt tuning for vision-language models by employing a three-ingredient approach ZOMP combines a crossmodal low-rank reparameterization that ties the two branches through a shared factor and keeps the effective search dimensionality small, a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, and a budgetindexed rank schedule that unlocks capacity as the query budget is spent This structure allows it to tune deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation.

Cross-Modal Low-Rank Prompt Parameterization

The core mechanism involves a cross-modal low-rank factorization that ties the vision and text prompt at each layer to a shared factor This is achieved by generating the prompts as the product of a small shared factor and a modality-specific factor, where P(l)v = U(l)V(l)v and P(l)t = U(l)V(l)t The trainable vector theta collects these factors, giving r(T +dv+dt) variables per layer and D = L r(T + dv + dt) in total. This factorization turns the rank into a handle the optimizer can grab, as component k occupies its own coordinates of theta, allowing the rank schedule to perturb only the first few components initially. Furthermore, this shared factor U(l) ties the two modalities together so that a single perturbation moves both coherently and exposes the rank as a knob.

Stabilized ZO with Progressive Rank Expansion

The optimization process utilizes simultaneous perturbation stochastic approximation (SPSA) to form an estimate by evaluating the loss at symmetric random offsets of the current iterate To stabilize these noisy SPSA estimates, ZOMP employs a gradient-correction momentum term, referred to as SPSA-GC (Oh et al. 2023) This momentum is defined by mk = βmk−1 +∇Lˆ (θk; Bk), and the update rule is θk+1 = θk − ηk∇Lˆ (θk; Bk) + βmk. The budgetindexed rank schedule turns the rank knob by masking perturbations to the first rj rank components when the query budget fraction q/Q is less than or equal to qj, effectively searching a small subspace first and widening it only once the optimizer has a direction worth expanding.

Empirical Results

Across 13 vision-language benchmarks under a matched 5,000-query budget, ZOMP consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency Table 1 reports that ZOMP attains the best average accuracy at 68.2%, 6.0 points ahead of ZIP at 62.2%. The gains over ZIP are largest on the specialized, non-natural-image domains where CLIP’s zeroshot prior is weakest, such as EuroSAT (+15.9), CLEVR (+10.3), DTD (+8.6), and SVHN (+8.1). In base-to-new generalization, ZOMP reaches an HM of 71.2%, exceeding ZIP’s 67.5% by 3.7 points and every other baseline by at least 6.6 points.

Component Ablation and Efficiency

Component ablation studies confirm that the two training mechanisms are complementary, as neither component improves meaningfully on its own The cross-modal low-rank factorization is the single largest step, lifting the average to 67.2% compared to a deep prompt optimized directly with SPSA without factorization, which reached 65.6%. The rank schedule alone gives 66.9%, and momentum alone gives 67.3%, but combining both lifts the average to 68.2%, a full point over the backbone and clearly ahead of either component in isolation. Regarding efficiency, ZOMP reaches its target accuracy in the fewest queries despite having the most trainable parameters (46.2k, against ZIP’s 0.4k) This is because ZOMP avoids searching the full parameterization all at once by expanding the rank gradually while momentum accumulates a stable descent direction before the noisier full-rank phase.

Conclusion

ZOMP makes deep vision-language prompting viable without backpropagation by structuring the search space via cross-modal low-rank factorization and stabilizing the noisy SPSA estimate with gradient-correction momentum, which are shown to be complementary mechanisms Across 13 vision-language benchmarks, ZOMP reaches the best few-shot accuracy and the fewest queries to target of any BP-free method, and it generalizes better than prior work across base-to-new, cross-dataset transfer, and outofdistribution settings. The results demonstrate that jointly exploiting multimodality and low-rank structure is an effective route to practical, queryefficient BP-free prompt tuning. The core idea is to stop optimizing the prompt matrices directly and instead search a cross-modal low-rank factorization, which ties the vision and text prompts at every layer to a shared low-dimensional factor. On top of this parameterization, gradient-correction momentum stabilizes the noisy SPSA estimate and a budgetindexed rank schedule holds the effective search space small early in training and widens it only once a reliable direction has formed. The controlled component ablation confirms that the two training mechanisms are complementary and only effective together. The paper concludes that ZOMP is a backpropagation-free method that makes deep multimodal prompt tuning practical under a realistic query budget.

--- Page 1 ---

The gist: ZOMP proposes a query-efficient, fully forward-only method that tunes deep prompts in both vision and text branches of frozen CLIP models using simultaneous perturbation stochastic approximation to achieve state-of-the-art performance under realistic query budgets.

Cross-Modal Low-Rank Prompt Parameterization

The core mechanism involves a cross-modal low-rank factorization that ties the vision and text prompt at each layer to a shared factor This is achieved by generating the prompts as the product of a small shared factor and a modality-specific factor, where P(l)v = U(l)V(l)v and P(l)t = U(l)V(l)t The trainable vector theta collects these factors, giving r(T +dv+dt) variables per layer and D = L r(T + dv + dt) in total. This factorization turns the rank into a handle the optimizer can grab, as component k occupies its own coordinates of theta, allowing the rank schedule to perturb only the first few components initially.

1. Bold header: Cross-modal low-rank prompt parameterization

ZOMP’s cross-modal low-rank factorization ties the vision and text prompt at each layer to a shared factor which allows a single update to move both coherently and exposes the rank as a knob. This enables a single perturbation of U(l) displaces the vision and text prompts coherently, leading to an optimization path that is amortized across both encoders, and the two prompt sets are constrained to a common tokenweighting pattern rather than drifting apart.

2. Bold header: Budget-indexed progressive rank schedule

The method employs a budgetindexed schedule that turns this knob, searching a small rank-1 subspace first and widening it only once the optimizer has a direction worth expanding, ensuring that the search space actually starts small and grows over training and avoiding the waste of queries by keeping the search coarse-to-fine.

3. Bold header: Stabilized zeroth-order optimization

ZOMP utilizes a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, which prevents updates from canceling each other, ensuring that the optimizer keeps descending along the direction found in the low-variance phase while the running average smooths out the added noise.

Prompt tuning adapts vision--language models with few trainable parameters, but computing their gradients still requires backpropagation through frozen encoders. We introduce ZOMP, a forward-only method for deep multimodal prompt tuning under a limited loss-query budget. ZOMP combines shared prompt factors with a budget-indexed rank schedule and gradient-correction momentum. Early updates search a rank-one subspace; later updates activate additional components without discarding the existing optimizer state. With 5,000 training-loss queries, ZOMP achieves 68.2% average 16-shot accuracy across 13 datasets, improving on ZIP by 6.0 points, alongside gains in base-to-new and distribution-shift generalization. On five image--text retrieval datasets, it outperforms four derivative-free baselines at both training sizes, improving average bidirectional Recall@1 over BPT-VLM by 2.63 points. An exploratory five-benchmark language diagnostic comparison examines caption discrimination beyond the retrieval galleries. Component, modality, depth, and budget studies characterize the method's benefits. ZOMP requires an internal-prompt inference interface and reduces allocated training memory relative to a matched-architecture first-order reference.