LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

The gist The LAMP framework proposes a training-free method that combines global and regional information from multiple encoder layers to improve few-shot vision-language adaptation.

How it works

LAMP introduces a training-free framework that combines global and regional information from multiple encoder layers using independently fitted kernel predictors LAMP, Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision–Language Adaptation, page 1. A single forward pass per view supplies all descriptors A single forward pass per view supplies all descriptors, page 1. Global and support-calibrated regional kernel predictors independently fit the same label residuals Global and support-calibrated regional kernel predictors independently fit the same label residuals, page 1. Labeled validation data select their mixture in class-score space, preserving native feature dimensions without learned projections Labeled validation data select their mixture in class-score space, preserving native feature dimensions without learned projections, page 1. LAMP uses complementary information from several layers without gradient optimization or learned feature projections LAMP uses complementary information from several layers without gradient optimization or learned feature projections, page 1.

Key Components and Mechanism

The core idea is to combine predictions from different layers in a shared class-score space, where each predictor outputs a score for the same set of classes Different layers provide different measures of image similarity, but each predictor outputs a score for the same set of classes. The method leverages the fact that predictions can be combined while preserving the original feature dimensions Fitting these predictors separately also allows their outputs to be reused when selecting the mixture, so each candidate mixture does not require a new fit.

The process involves several steps for a single query q, where frozen CLIP encodes visual representations and regional descriptors are extracted from intermediate layers A single encoder pass per view yields a global descriptor and regional descriptors from three layers of a Vision Transformer (ViT) or ResNet. The global predictor and three regional predictors independently fit the same label residual, which is defined as the difference between the support labels and CLIP’s initial scores Each regional kernel is calibrated against the support set Labeled validation data select the mixture of global and regional corrections.

LAMP-F Extension

The extension, LAMP-F, adapts text-anchored class prototypes while retaining the selected kernels and mixture weights LAMP-F further adapts text-anchored class prototypes while retaining the selected kernels and mixture weights, page 1. It uses a cached response to update the fixed-kernel residual predictions exactly LAMP-F uses a cached response to update the fixed-kernel residual predictions exactly, page 1. This is achieved by accounting for both the initial scores and residual corrections through a cached linear response, avoiding repeated matrix factorization during prototype evaluation LAMP-F further adapts text-anchored class prototypes while retaining the selected kernels and mixture weights, page 1.

Performance and Contributions

LAMP achieves state-of-the-art mean accuracy among the compared training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots LAMP achieves state-of-the art mean accuracy among the compared training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots, page 1. LAMP improves over a paired global-feature-only baseline by an average of 1.14 and 1.59 percentage points on ViT and ResNet-50 Averaged across five support budgets, LAMP improves over a paired global-feature-only baseline by an average of 1.14 and 1.59 percentage points on ViT and ResNet-50 page 2. LAMP achieves state-of-the art mean accuracy among the compared training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots, page 1.

The contributions include:

We introduce LAMP, a training-free method that combines global and regional information from multiple encoder layers using independently fitted kernel predictors

We develop LAMP-F, which combines prototype adaptation with the same global and regional predictors

We demonstrate state-of-the art mean accuracy in the published training-free RN50 comparison at 4, 8, and 16 shots across eleven datasets

Analysis of Gains

Controlled experiments identify multi-layer integration as the main source of improvement Controlled experiments identify multi-layer integration as the main source of improvement, page 1. The paper demonstrates that combining layer-specific predictions can exploit complementary features from frozen encoders to improve few-shot classification These findings show that combining layer-specific predictions can exploit complementary features from frozen encoders to improve few-shot classification, page 1.

The analysis of performance gains shows that prototype adaptation and native-layer integration thus benefit different tasks: the former adjusts class directions, while the latter changes how support evidence is used to correct the initial CLIP scores Prototype adaptation benefits a different set of tasks, with its largest 4-shot gains occurring on SUN397, at 5.620 points with ViT and 6.715 with RN50 LAMP-F further improves the means to 68.83 for RN50 LAMP-F further improves the means to 68.83 for RN50, page 12.

Computational Considerations

The computational cost is noted, stating that LAMP increases single-image latency from 14.22 to 64.83 ms on ViT-B/16 LAMP increases single-image latency from 14.22 to 64.83 ms on ViT-B/16, page 20. Dense fitting requires O(N 2) storage and O(N 3) factorization per branch Dense fitting requires O(N 2) storage and O(N 3) factorization per branch, page 20. LAMP-F also stores prototype responses, increasing peak allocated GPU memory to 8.56 GiB with ViT LAMP-F also stores prototype responses, increasing peak allocated GPU memory to 8.56 GiB with ViT, page 20.

The paper concludes that LAMP provides a way to combine features of different dimensions without learning a shared projection Beyond these accuracy gains, LAMP provides a way to combine features of different dimensions without learning a shared projection, page 22. The separation of branch fitting, score fusion, and prototype updates offers a foundation for studying task-specific layer selection and developing more scalable residual prediction methods The separation of branch fitting, score fusion, and prototype updates offers a foundation for studying task-specific layer selection and developing more scalable residual prediction methods, page 22.

Future Directions

The work examines adaptation across encoder architectures, depths, and patch sizes To examine adaptation across encoder architectures, depths, and patch sizes, page 18. The results demonstrate that native-layer corrections remain useful under both distribution shifts LAMP improves accuracy in all twelve combinations of seed, target dataset, and backbone LAMP improves accuracy in all twelve combinations of seed, target dataset, and backbone, page 20.

The paper also tests transfer to new domains without fitting on target data and measures deployment costs The paper also tests transfer to new domains without fitting on target data and measures deployment costs, page 18. LAMP improves accuracy in all twelve combinations of seed, target dataset, and backbone, page 20.

The final results show that native-layer corrections remain useful under both distribution shifts LAMP improves accuracy in all twelve combinations of seed, target dataset, and backbone under distribution shifts LAMP improves accuracy in all twelve combinations of seed, target dataset, and backbone under distribution shifts.

The paper is supported by the Young Scientific and Technological Talent Support Program under the Taizhou Fengcheng Talent Plan This work was supported by the Young Scientific and Technological Talent Support Program under the Taizhou Fengcheng Talent Plan, page 23. The authors used OpenAI ChatGPT/Codex to assist with language editing, document organization, experimental-code preparation, and consistency checks against the recorded experiments During manuscript preparation, the authors used OpenAI ChatGPT/Codex to assist with language editing, document organization, experimental-code preparation, and consistency checks against the recorded experiments. The paper is supported by the Young Scientific and Technological Talent Support Program under the Taizhou Fengcheng Talent Plan This work was supported by the Young Scientific and Technological Talent Support Program under the Taizhou Fengcheng Talent Plan, page 23.

1. Bold header: Training-Free Multi-Layer Feature Fusion

This system can combine global and regional information from multiple encoder layers to create a richer representation, which is shown to improve performance by leveraging complementary features from frozen encoders.

2. Bold header: Independent Kernel Prediction and Score Fusion

The framework allows for Global and support-calibrated regional kernel predictors independently fit the same label residuals, enabling the system to combine predictions in class-score space while preserving native feature dimensions without learned projections.

3. Bold header: Prototype Adaptation with Cached Response

The LAMP-F extension enables adaptation by using a cached response to exactly update both the initial class scores and residual corrections when prototypes change, allowing for efficient and exact updates to class representations.