Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling

arXiv:2610.22267 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Beyond Pairwise Repulsion".

Jane: The gist The proposed DetNO method recovers suppressed modes in diffusion models by formulating diversity as an RKHS volume maximization objective,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at "Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling," what’s really sticking with me is how they moved beyond simple pairwise repulsion to this set-level functional.

Jane: It feels like they stopped trying to solve the problem image by image and started solving it for the entire batch at once, which is a fundamental shift in thinking for how we approach distribution sampling in these models.

Lu: The main implication is that we can recover degrees of freedom left unexploited by standard sampling methods because this log-determinant objective captures redundancy at every order simultaneously.

Meng: For deployment, the fact that the method is controllable with a single hyperparameter lambda for the diversity quality trade-off, and it doesn't require training changes, makes it appealing for integrating into existing pipelines.

Tom: It’s about being able to select exactly how much diversity you want versus how much fidelity you need without having to modify the underlying sampling process or train new components.

Jane: The authors confirm that across SD1 point 5, SDXL, and SD3, this approach delivers leading intra-batch diversity and achieves the best downstream accuracy when comparing different batch objectives against each other on both benchmarks <ref:2610.22267#pg1>.

Lu: Their finding that this method outperforms InfoNCE and attains the best log-determinant accuracy suggests it’s a robust way to capture the full distributional breadth of the initial noise space.

Meng: I think for production, this means we have a new objective function we can swap into our sampling loop to see if we can generate better synthetic data for our specific use cases.

Tom: The core idea is that this collective volume maximization approach provides a controllable way to get image sets that are not only more varied but measurably more useful for downstream tasks like classification in a low-shot setting.

Conclusion: Tom: So we've been looking at DetNO, this paper titled "Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling." Essentially, they’re saying that instead of just pushing individual noisy images apart one by one, you should look at the whole batch together.

Jane: Right. The title itself says "Beyond Pairwise Repulsion." It suggests that what we've been doing with sum-of-pairwise objectives isn't enough to get the full range of possibilities out of a diffusion model.

Lu: Exactly. They formulate diversity as this RKHS volume maximization objective on the initial noise. Think about it like this: they’re maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. That's a set-level functional, not just looking at two images at a time.

Meng: From a deployment standpoint, that sounds complex to set up initially. But if it works, it means we can get better synthetic data augmentation without retraining the entire sampler or changing how we run the inference step itself.

Tom: That’s what I find compelling. The authors show that this collective volume maximization is self-regulating; when the batch gets too redundant, that log-determinant objective pushes back harder, and once you hit a good level of diversity, it relaxes a bit. It's like having built-in feedback for distribution quality.

Jane: And they proved it with results across three different diffusion backbones: SD1 point five SDXL, and even the newer SD3. They found that this method gives leading intra-batch diversity metrics like VendiScore and Mean Self-Similarity compared to other methods like InfoNCE.

Lu: The key result they highlight is that this log-determinant objective actually attains the best downstream accuracy in a few-shot classification experiment, which is pretty significant for synthetic data use cases. It shows that broader mode coverage truly helps the model learn better features when you only have a few examples to start with.

Tom: So, if we boil it down, they found that this set-level approach beats pairwise repulsion and achieves better results than other log-determinant based aggregators too. It’s a two-term objective with one hyperparameter lambda controlling the trade off between diversity and fidelity without any training.

Jane: The implication is that for anyone building synthetic data pipelines, you can now select exactly how diverse your generated set needs to be versus how realistic those images need to look, all by tuning that single lambda value.

Lu: It’s a powerful way to recover those suppressed modes in the diffusion process without needing complex modifications to the sampling schedule or any new training data.

Meng: I'm still focused on scaling this up and seeing if it introduces any weird artifacts when we push batch sizes really high, but the initial evidence suggests it’s very clean.

Tom: So, this paper shows a way to control image diversity by optimizing the volume of the noise space itself, which opens up new possibilities for making AI-generated data more useful and varied.

Jane: That leads us next to how this applies when we look at other ways to measure distribution quality in these models.

Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Sungho Koh, Dong-Jin Kim

Hanyang University

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: ACCV 2026, oral presentation

Code: https://github.com/mobled37/DetNO

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The proposed DetNO method recovers suppressed modes in diffusion models by formulating diversity as an RKHS volume maximization objective, which outperforms pairwise repulsion methods and

Key concepts

Classifier-Free Guidance (CFG) Limitation: Standard guidance techniques concentrate sampling toward high-density modes, effectively collapsing the visual support onto a narrow region of the image manifold. This limits the model's ability to generate varied outputs, which is detrimental for tasks requiring broad distributional coverage.
RKHS Volume Maximization: Diversity is formulated as maximizing the volume a batch collectively occupies in a Reproducing Kernel Hilbert Space (RKHS). This is achieved by maximizing the log-determinant of an RBF Gram matrix calculated from one-step Tweedie predictions. It penalizes redundancy across all elements in the batch simultaneously.
Anchor-Based Fidelity Term: A constraint term ensures that the optimized noise vectors remain close to plausible regions by constraining their displacement from reference anchors. This fidelity term is incorporated via Lagrangian relaxation, allowing a single hyperparameter $\lambda$ to control the trade-off between maximizing diversity and maintaining image quality.
Self-Attenuating Log-Determinant: The log-determinant objective is self-regulating: it amplifies repulsion when the batch is most redundant and naturally relaxes as diversity is achieved. This mechanism separates samples without needing per-step intervention, proving that optimizing the noise at time $t=T$ is sufficient for diverse results.

Terminology

Accuracy — 2× in this paper · explained in 1 episode
Data Augmentation — 2× in this paper · explained in 7 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Diffusion Model — 1× in this paper · explained in 10 episodes
Diffusion Models — 1× in this paper · explained in 9 episodes
Fidelity — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed DetNO method recovers suppressed modes in diffusion models by formulating diversity as an RKHS volume maximization objective, which outperforms pairwise repulsion methods and provides statistically significant accuracy gains in few-shot classification.

How it works

The paper addresses the limitation of Classifier-Free Guidance (CFG) which concentrates sampling toward high-density modes, collapsing the effective support onto a narrow region of the image manifold <ref:2610.22267#pg6>.
DetNO recovers this freedom by treating a batch as a whole and formulating diversity as RKHS volume maximization on the initial noise, maximizing the logdeterminant of an RBF Gram matrix over one-step Tweedie predictions <ref:2610.22267#pg5>.
This objective is paired with an anchor-based fidelity term via Lagrangian relaxation, controlled by a single hyperparameter λ governing the diversity–quality trade-off <ref:2610.22267#pg12>.
The loss function is defined as L = Ldiv + λ · Lfid, where the diversity term minimizes - log det(S + ϵI) and the fidelity term is an anchor-based displacement constraint <ref:2610.22267#pg13>.

Diversity Maximization

The diversity term quantifies diversity as the volume a batch collectively occupies, a set-level functional that penalizes redundancy at every order simultaneously <ref:2610.22267#pg14>. This is realized through the log-determinant of an RBF Gram matrix over one-step Tweedie predictions <ref:2610.22267#pg15>. The paper demonstrates that this objective is self-regulating: repulsion amplifies when the batch is most redundant and relaxes as diversity is achieved <ref:2610.22267#pg16>.

Fidelity Constraint

The fidelity term ensures that optimized noises remain near plausible regions by imposing a constraint on displacement from reference anchors, defined by the inequality zˆ 0T i - zˆ 0T i,ref squared ≤ δ <ref:2610.22267#pg19>. The final objective is derived as L = -log det (S + ϵI) + λ Σ zˆ 0T i - zˆ 0T i,ref squared, where λ ≥ 0 controls the trade-off <ref:2610.22267#pg12>.

Empirical Validation and Results

DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity across three backbones (SD1.5, SDXL, SD3) and both samplers (DDIM and Euler) <ref:2610.22267#pg3>. In downstream few-shot classification experiments, the broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime <ref:2610.22267#pg11>. DetNO consistently outperforms DDIM and CNO across both datasets, with gains most pronounced in the low-data regime <ref:2610.22267#pg13>.

Key Findings

The log-determinant instead self-attenuates as volume is restored (Proposition 1), separating samples without displacing the modes themselves

This approach validates that optimizing the Tweedie prediction at t=T is sufficient to produce diverse final images without per-step intervention <ref:2610.22267#pg22>. The method's robustness is confirmed by showing that its advantage over CNO widens as the batch size grows, consistent with the all-order argument <ref:2610.22267#pg18>.

The log-determinant attains the best intra-batch diversity, and both RBF-based aggregators outperform InfoNCE

This framework successfully recovers degrees of freedom in the initial noise that standard sampling and pairwise objectives leave unexploited <ref:2610.22267#pg24>. The results confirm that noise scale manipulation is not a viable mechanism for diversity enhancement <ref:2610.22267#pg24>.

The DetNO framework provides a controllable quality-diversity trade-off, allowing practitioners to select configurations based on desired outcomes <ref:2610.22267#pg12>. This method produces image sets that are not only more varied but measurably more useful <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>. The final objective is a single scalar measuring the collective volume a batch occupies, penalizing redundancy at every order up to B simultaneously <ref:2610.22267#pg14>. This entire process is achieved without training or sampler modification <ref:2610.22267#pg12>. The method's performance is demonstrated across three backbones and two samplers, confirming consistent gains with a controllable quality-diversity trade-off and statistically significant downstream improvements in few-shot classification <ref:2610.22267#pg11>.

Improvements for AI systems

Strict Mode Diversity via DetNO Objective Formulation: The improved system can generate image sets that are not only more varied but measurably more useful, by treating diversity as a set-level rather than pairwise problem: constrained RKHS volume maximization on initial noise. This is achieved by formulating the objective as the log-determinant of an RBF Gram matrix over one-step Tweedie predictions, which recovers suppressed degrees of freedom in initial noise that standard sampling cannot access.
Controllable Quality-Diversity Trade-off: The improved system allows users to tune the balance between diversity and quality via a single hyperparameter "λ without training or sampler modification. This is realized through the objective: L = Ldiv + λ · Lfid, where Ldiv maximizes the RKHS volume and Lfid uses an anchor-based fidelity term" to keep optimized noises near plausible regions, ensuring that diversity gains are achieved while preserving prompt fidelity.
Self-Regulating Repulsion Mechanism: The improved system features a mechanism where repulsion amplifies when the batch is most redundant and relaxes as diversity is achieved, which is quantified by Proposition 1 showing that the gradient ratio between the log-determinant loss and pairwise objectives self-attenuates. This ensures that the repulsion intensifies as the batch collapses and relaxes once volume is restored, leading to a more robust recovery of all modes, as demonstrated in qualitative comparisons like Fig. 3.
Enhanced Downstream Utility for Synthetic Data Augmentation: The improved system translates its intra-batch diversity into statistically significant gains for downstream tasks. It achieves statistically significant accuracy gains for synthetic data augmentation in the low-shot regime, with DetNO reaching the best results on few-shot classification benchmarks by recovering broader mode coverage.

Abstract

Strong guidance in text-to-image (T2I) diffusion models concentrates sampling toward high-density modes of the conditional distribution, collapsing independently sampled noise vectors onto nearly identical visual prototypes. This limits utility for tasks that demand distributional breadth, most critically synthetic data augmentation. Diversity, however, is a collective property of the whole batch: a set can preserve every pairwise distance yet still collapse onto a low-dimensional subspace. Existing sum-of-pairwise noise-optimization and repulsion objectives respond only weakly to this higher-order failure. We propose DetNO (Determinant Noise Optimization), which formulates diverse generation as RKHS volume maximization on the initial noise, maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. This set-level functional equals the squared batch volume; its repulsion is strongest when the batch is most collapsed and self-attenuates as diversity is achieved. An anchor-based fidelity term keeps the optimized noises near plausible regions, and Lagrangian relaxation gives a two-term objective with a single hyperparameter λ governing the diversity-quality trade-off, without training or sampler modification. Across three backbones (SD1.5, SDXL, SD3), DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, and a downstream few-shot classification experiment confirms that broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime. Fixing the pipeline and varying only the batch objective, both RBF-based aggregators outperform InfoNCE, and the log-determinant attains the best downstream accuracy on both benchmarks. Its diversity advantage over InfoNCE widens as the batch grows. Code is available at https://github.com/mobled37/DetNO.