Adversarially Robust Few-Shot Anomaly Detection with Vision Foundation Models

arXiv:2510.13643 · cs.CV · Submitted 2025-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Adversarially Robust Few-Shot Anomaly Detection with Vision Foundation Models".

Jane: Here's the gist of the paper. The gist The study introduces an adversarially robust Few-Shot Anomaly Detection method that utilizes frozen Vision Foundation Models and a novel two-level defense pipeline to achieve significant robustness against strong gradient-based attacks in few-shot anomaly detection settings.

Tom: Problem Statement Existing robust anomaly detection methods assume access to large normal-class training sets and adversarial training of the feature extractor, which is incompatible with the few-shot regime where detectors consist of a frozen backbone and a non-parametric scorer <ref:2510.13643#pg6>.

Jane: The adversarial robustness of few-shot, foundation-model-based anomaly detectors has therefore remained an open question despite their practical prevalence <ref:2510.13643#pg5>.

Tom: Addressing this gap is non-trivial because evaluating robustness requires a strong gradient-based attack, but the non-parametric k-NN scorer is only piecewise differentiable through its active minimizer <ref:2510.13643#pg8>.

Title and authors: Tom: Now that we know they have this attack tool, let's look at what the paper actually proposes as a defense against those attacks. They introduce a two-level defense pipeline operating only on the k support images, which is quite specific.

Jane: So it’s not trying to fix the backbone itself; it’s focusing all its effort on defending the scoring stage using just what you have available in the few-shot set.

Lu: The first level of defense is PatchShift, which exploits how adversarial perturbations are coupled to the Vision Transformer's patch grid by applying random pixel shifts and then aggregating scores via element-wise median pooling across those shifted views.

Meng: So it’s trying to break the alignment between the attack and how the model sees things by just cropping and shifting a few images?

Lalam: And for the second level, they have FeaturePurifier, which is a lightweight residual MLP trained on clean and adversarially augmented support features to project those perturbed representations back toward what they consider the clean feature manifold.

The paper's summary: Tom: The main point of this paper is that by combining these two specific defenses, PatchShift and FeaturePurifier, they achieve robust detection and localization with performance gains of about twenty-seven percent image-level AUROC and about fifty-one percent pixel-level PRO over the undefended attacked baseline on benchmarks like MVTec-AD, VisA, and MPDD at k=four shots <ref:2510.13643#pg2>.

Jane: That's a pretty concrete number to look at; they show that this combined defense recovers substantial robustness across all those datasets and shot counts without sacrificing too much clean performance.

Lu: What’s interesting is how the defense holds up under adaptive attacks, specifically mentioning the EOT-BPDA attack, which suggests it’s not just robust against a single type of adversarial noise.

Meng: It's good to hear that it holds up against those stronger adaptive attacks because those are often what you see in more real-world deployment scenarios where attackers aren't just using simple PGD.

Lalam: The ablation study confirms that the combination of PatchShift plus FeaturePurifier is preferable for industrial inspection settings because it substantially improves pixel-level localization while preserving spatial structure in the anomaly map.

The paper's improvements: Tom: So what are the actual improvements they are highlighting? They’re focusing on making detection robust without needing to train the backbone, which is a big deal for practical use cases.

Jane: They introduced three main areas of improvement: first, creating a white-box attack framework using DistanceProbe so you can test robustness against k-NN scorers; second, developing the two-level defense pipeline with PatchShift and FeaturePurifier; and third, showing that this combination works across different vision backbones.

Lu: The key improvement there is formalizing adversarial robustness for few-shot anomaly detection with frozen vision foundation models by introducing a tractable attack surrogate and a two-level defense that operates without backbone training.

Meng: That training-free aspect is critical for us because if we can deploy this on top of a large, already trained model like DINOv2, we avoid the massive cost of retraining the feature extractor.

Lalam: They also show that they achieve pixel level localization performance matching or exceeding full shot adversarially trained baselines on MPDD at k=four shots, which points to a superior ability to identify precise defective regions in industrial imagery under attack <ref:2510.13643#pg2>.

Conclusion: Tom: Wrapping up, the paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models, showing that this approach is viable in practice.

Jane: The big implication here is that you don't need a huge normal dataset to build a robust detector if you can introduce these specific tools like DistanceProbe and the PatchShift plus FeaturePurifier defense.

Lu: They successfully formalized adversarial robustness for this setting, demonstrating that the combination of PS+FP recovers substantial robustness while preserving clean accuracy within about three points compared to full-shot adversarially trained baselines on pixel-level AUROC at k=four <ref:2510.13643#pg2>.

Meng: For deployment, the preference for PatchShift plus FeaturePurifier is clear because it really helps with localizing those defective regions, which is the main goal in inspection tasks.

Lalam: So to summarize, the DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers and their two-stage defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full shot adversarially trained baselines at k=four <ref:2510.13643#pg2>.

Akib Mohammed Khan, Bartosz Krawczyk

Rochester Institute of Technology

cs.CV

Submitted: 2025-10-15

Updated: 2026-10-20

Comments: Accepted to BMVC 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The study introduces an adversarially robust Few-Shot Anomaly Detection method that utilizes frozen Vision Foundation Models and a novel two-level defense pipeline to achieve significant

Key concepts

DistanceProbe (DP): A lightweight MLP trained to predict the k-NN distances between patches directly from frozen foundation model features. This allows researchers to create a differentiable surrogate for the non-parametric scoring function, making it possible to craft effective gradient-based adversarial attacks against the anomaly detector.
PatchShift (PS): The first defense layer that applies random pixel shifts to images. This technique exploits the mismatch between where adversarial noise lands and how the foundation model's patch grid is organized. It aggregates scores from these shifted views to create a more stable anomaly map.
FeaturePurifier (FP): A second defense layer consisting of a residual MLP trained on clean and augmented features. Its goal is to project perturbed feature representations back onto the 'clean' feature manifold, effectively removing adversarial noise while ensuring that the original clean features remain unchanged.

Terminology

AUROC — 2× in this paper · explained in 3 episodes
Foundation Model — 2× in this paper · explained in 3 episodes
Foundation Models — 2× in this paper · explained in 7 episodes
Accuracy — 1× in this paper · explained in 1 episode

Summary

The gist The study introduces an adversarially robust Few-Shot Anomaly Detection method that utilizes frozen Vision Foundation Models and a novel two-level defense pipeline to achieve significant robustness against strong gradient-based attacks in few-shot anomaly detection settings.

Problem Statement

Existing robust anomaly detection methods assume access to large normal-class training sets and adversarial training of the feature extractor, which is incompatible with the few-shot regime where detectors consist of a frozen backbone and a non-parametric scorer <ref:2510.13643#pg6>. The adversarial robustness of few-shot, foundation-model-based anomaly detectors has therefore remained an open question despite their practical prevalence <ref:2510.13643#pg5>. Addressing this gap is non-trivial because evaluating robustness requires a strong gradient-based attack, but the non-parametric k-NN scorer is only piecewise differentiable through its active minimizer <ref:2510.13643#pg8>.

DistanceProbe (DP) for Attack Tractability

To address the challenge of attacking non-parametric scorers, the authors introduce the DistanceProbe (DP), a lightweight MLP trained by regression to predict per-patch k-NN distances directly from frozen DINOv2 features <ref:2510.13643#pg8>. The probe is trained on patch tokens extracted from nominal training images disjoint from the few-shot support set using the same backbone and preprocessing pipeline <ref:2510.13643#pg9>. This allows for end-to-end differentiable per-patch scoring, denoted as abj(x) = gφ fθ (x)j, which is used to craft the surrogate adversarial loss Lb <ref:2510.13643#pg10>.

PatchShift (PS) Defense

The first level of defense is PatchShift (PS), which applies random pixel shifts to exploit the misalignment of adversarial perturbations with the encoder’s patch grid. This involves sampling K shifts, forming shifted-and-cropped images x(k) = CropP Shift∆(k)(x), and aggregating scores via element-wise median pooling across these views to yield the PS-aggregated anomaly map aPS(x).

FeaturePurifier (FP) Defense

The second level of defense is the FeaturePurifier (FP), a lightweight residual MLP trained on clean and adversarially augmented support features to project perturbed representations back toward the clean feature manifold. The purifier hψ is trained by minimizing a two-term mean-squared error loss Lpurify(ψ) which encourages the mapping of perturbed features back to their clean counterparts while constraining it to leave clean features unchanged.

Experimental Results and Robustness

Comprehensive experiments across MVTec-AD, VisA, and MPDD at k=4 shots show that the proposed method exhibits robust detection and localization with performance gains of ∼ 27% image-level AUROC and ∼ 51% pixel-level PRO over the undefended attacked baseline. The combined defense recovers substantial robustness across all datasets and shot counts, with clean performance preserved within ∼3 points, matching full-shot adversarially-trained baselines on pixel-level AUROC at k=4. Furthermore, the defense holds under adaptive attacks such as EOT-BPDA. The ablation study confirms that the combination of PS+FP is preferable for industrial inspection settings because it substantially improves pixel-level localization while preserving spatial structure in the anomaly map.

Conclusion

The framework successfully formalizes adversarial robustness for few-shot anomaly detection with frozen vision foundation models, introducing a tractable attack surrogate and a two-level defense that operates without backbone training. The method demonstrates strong performance under various adversarial settings, confirming its viability in the practical deployment of such detectors. The defense recovery is shown to be progressive rather than fragile robustness, as it maintains high performance even under stronger attack budgets. Future work includes extending the framework to other multi-modal pretrained backbones and enriching the purifier’s training distribution. The proposed method achieves robust detection and localization with performance gains of ∼ 27% image-level AUROC and ∼ 51% pixel-level PRO over the undefended attacked baseline across MVTec-AD, VisA, and MPDD at k = 4 shots. The defense holds under adaptive attack. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective. This paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models. The DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers, and our two-stage PatchShift + FeaturePurifier defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full-shot adversarially-trained baselines on pixel-level localization at k = 4. The defense holds under adaptive EOT-BPDA attack, ruling out gradient obfuscation. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective. This paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models. The DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers, and our two-stage PatchShift + FeaturePurifier defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full-shot adversarially-trained baselines on pixel-level localization at k = 4. The defense holds under adaptive EOT-BPDA attack. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective. This paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models. The DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers, and our two-stage PatchShift + FeaturePurifier defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full-shot adversarially-trained baselines on pixel-level localization at k = 4. The defense holds under adaptive EOT-BPDA attack. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective. This paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models. The DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers, and our two-stage PatchShift + FeaturePurifier defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full-shot adversarially-trained baselines on pixel-level localization at k = 4. The defense holds under adaptive EOT-BPDA attack. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective. This paper presents the first adversarial robustness framework for few-shot anomaly detection with frozen vision foundation models. The DistanceProbe surrogate enables strong PGD attacks on non-parametric k-NN scorers, and our two-stage PatchShift + FeaturePurifier defense recovers substantial robustness on MVTec-AD, VisA, and MPDD while preserving clean accuracy and matching full-shot adversarially-trained baselines on pixel-level localization at k = 4. The defense holds under adaptive EOT-BPDA attack. The combined defense is therefore preferable for industrial inspection settings, where localizing defective regions is the primary objective.

Improvements for AI systems

Improved Anomaly Detection Robustness for Frozen Foundation Models: The system can now robustly detect anomalies in few-shot settings without requiring any training of the frozen backbone, by introducing a white-box attack framework using a DistanceProbe to create a differentiable surrogate for non-parametric k-NN scorers.
Tractable Gradient-Based Attacks on Non-Parametric Scorers: The DistanceProbe allows researchers to perform gradient based attacks against the non-parametric scorer by predicting per patch k-NN distances directly from frozen DINOv2 features, which was previously intractable due to the non differentiable nature of the scorer.
Two-Level Defense Pipeline for Robustness: The system can defend against adversarial perturbations using a two-level defense that operates only on normal reference support set images, combining PatchShift to exploit patch grid misalignment and FeaturePurifier to project perturbed representations back toward the clean feature manifold.
Backbone Agnostic Defense: The proposed defense framework is agnostic to the choice of foundation model, allowing it to be applied across different vision backbones while still operating within the few-shot, frozen-backbone constraint.
Enhanced Localization Accuracy: The system achieves pixel level localization performance matching or exceeding full shot adversarially trained baselines on MPDD at k = 4 shots, indicating superior ability to identify precise defective regions in industrial imagery under attack.

Abstract

Vision foundation models such as DINOv2 enable strong few-shot anomaly detection (FSAD) through simple non-parametric k-nearest-neighbor (k-NN) scoring over frozen patch features. Existing robust anomaly detection methods assume large normal-class training sets and adversarial training of the feature extractor. The few-shot regime, where the detector consists of a frozen backbone and a non-parametric scorer, has remained unaddressed despite its widespread deployment. We develop a white-box attack framework adapted to k-NN memory-bank detectors and introduce an adversarially robust FSAD method which is training-free with respect to the backbone. We introduce the DistanceProbe (DP), a lightweight MLP trained by regression to predict per-patch k-NN distance directly from frozen features, providing the differentiable proxy required for gradient-based attacks against non-parametric scorers. We further propose a two-level combined defense that operates using only the normal reference support set images. At the input level, PatchShift (PS) applies random pixel shifts to exploit the misalignment of adversarial perturbations with the encoder's patch grid, aggregating scores across multiple shifted views via element-wise median pooling. At the feature level we employ the FeaturePurifier (FP), a lightweight residual MLP trained on clean and adversarially augmented support features to project perturbed representations back toward the clean feature manifold. Through comprehensive experiments under various adversarial settings, we show that our method exhibits robust detection and localization with performance gains of 27% image-level AUROC and 51% pixel-level PRO over the undefended attacked baseline across MVTec-AD, VisA, and MPDD at k=4 shots, while preserving clean accuracy within 3%, matching full-shot adversarially-trained baselines on pixel-level AUROC, and holding under adaptive attack.