Efficient quadratic entropy with distance sketches

arXiv:2610.11976 · stat.ML, cs.LG, math.ST, stat.CO, stat.TH · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Efficient quadratic entropy with distance sketches".

Jane: The gist A scalable method for approximating quadratic entropy using random feature embeddings and projections dramatically improves computational complexity for common distances like Euclidean and spherical geodesic cases,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We've talked about the core idea behind the paper "Efficient quadratic entropy with distance sketches," which is using random feature embeddings and projections to make approximating quadratic entropy much more computationally tractable for common distances.

Jane: They show that this approach works for any distribution p and any common distance d as long as that distance has negative type, page one <ref:2610.11976#pg1>.

Lu: The paper details efficient estimators for common distances of negative type, focusing specifically on squared Euclidean and cosine distances which yield trivially efficient computations.

Meng: They show that for ordinary Euclidean distance, an unbiased estimator euc N is computed by sampling IID vectors u from the standard normal distribution N(zero I m) and computing the sample mean of q euc(u), defined as pi over two sum j,k p j p k u T x(j) - u T x(k), page three <ref:2610.11976#pg3>.

Lalam: This estimator is shown to be unbiased, with a relative error bound of P euc N / Q - one at least epsilon at most two (-N epsilon two/pi), and they have a squared-Euclidean control variate with mean computable in O(mn), page two <ref:2610.11976#pg3>.

Tom: The paper also tackles the spherical geodesic distance, showing how to linearize it through random sign features s j(u) to get theta jk = pi/two Eu(one - s j s k), which leads to Theorem three point seven, page three <ref:2610.11976#pg3>.

Jane: This means that for the spherical geodesic distance matrix on an (m-one) -sphere, p T d p = pi/two (one - E

s two: ), and they show a corresponding unbiased estimator geo N with a relative error bound of P geo N - Q at least epsilon at most two (-8N epsilon two/pi two), page three.

Lu: The key mechanism here is transforming the distance matrix using the Cholesky factorization of G = -one/two J T d J, where the columns psi(j) of are those desired feature vectors, and inserting that into the formula for quadratic entropy, which gives p T dp = two X j p j psi(j) squared - X j p j psi(j) squared, page three.

Meng: This transformation is what makes the right side of equation (three) cheaper to compute than the left side's form suggests, which is a big deal for performance <ref:2610.11976#pg2>.

Lalam: And they prove that the Euclidean and cosine distances yield trivially efficient computations, page three <ref:2610.11976#pg2>.

Tom: So this framework shows how these feature embeddings can be used to make quadratic entropy calculations much faster for common distances like Euclidean and spherical geodesic ones.

Jane: It’s about using these projections to move away from the direct pairwise calculations which are slow, towards a more structured computation that scales better.

Conclusion: Tom: So wrapping up the discussion on "Efficient quadratic entropy with distance sketches," we've seen how this paper uses random feature embeddings to tackle quadratic entropy approximation for common distances.

Jane: The authors prove that these sketch-based estimators give unbiased estimates of quadratic entropy with error bounds that don't depend on the ambient dimension m, page two <ref:2610.11976#pg3>.

Lu: This independence from m is a key result, meaning you can use this method reliably even when the underlying structure has a very high dimension.

Meng: For someone who only listens to this show, it means they don't have to worry about the complexity exploding as the data gets more intricate in terms of features.

Lalam: This framework gives us a way to approximate Q = p T d p for arbitrary distributions p and common distances d of negative type, page one <ref:2610.11976#pg1,for arbitrary distributions $p$ and common distances $d$ of negative type>.

Tom: And it also demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count chi.

Jane: The work provides a framework for approximating Q = p T d p for arbitrary distributions p and common distances d of negative type, page one <ref:2610.11976#pg1,for arbitrary distributions $p$ and common distances $d$ of negative type>.

Lu: It shows that this is applicable to a very wide range of scenarios in bibliometric contexts, from simple dispersion to more complex interdisciplinary reach.

Meng: This paper gives us a way to approximate Q = p T d p for arbitrary distributions p and common distances d of negative type, page one <ref:2610.11976#pg1,for arbitrary distributions $p$ and common distances $d$ of negative type>.

Lalam: And the work provides a framework for approximating Q = p T d p for arbitrary distributions p and common distances d of negative type, page one <ref:2610.11976#pg1,for arbitrary distributions $p$ and common distances $d$ of negative type>.

Tom: It’s a solid contribution because it provides a practical method that can be applied to many real-world problems where speed matters.

stat.ML, cs.LG, math.ST, stat.CO, stat.TH

Submitted: 2026-10-08

Updated: 2026-10-09

Comments: Code for reproducing results in LaTeX comments; removed submission checklist from v1

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist A scalable method for approximating quadratic entropy using random feature embeddings and projections dramatically improves computational complexity for common distances like Euclidean and

Key concepts

Negative Type Distances: These are specific types of distance matrices where a certain mathematical condition holds true. A key property is that if a matrix satisfies this condition, its distances can be represented as the squared Euclidean distance between some feature vectors. This connection is fundamental to using random features for approximation.
Feature Embeddings and Projections: The paper uses random sign features (like sj(u) = sgn(uT x(j))) to linearize complex distances, such as the spherical geodesic distance. These projections transform the original problem into a simpler form involving Euclidean distances between the projected features, which are easier to estimate efficiently.
Amortization Techniques: This technique allows the method to reuse computations across different instances of 'p' (a distribution parameter). Instead of recomputing everything from scratch for every new instance, the method reuses a single precomputed product (U T X) and performs subsequent updates efficiently, drastically improving scaling for large datasets.
Quadratic Entropy Approximation: The goal is to estimate the quadratic entropy, which measures the dispersion or spread of data points according to a distribution 'p'. By relating this entropy to distances derived from random features, the method provides an unbiased estimator that captures this underlying semantic dispersion in real-world data.

Terminology

Papers — 34× in this paper · explained in 3 episodes
Embeddings — 10× in this paper · explained in 2 episodes
Benchmark — 5× in this paper · explained in 2 episodes
Generative AI — 3× in this paper · explained in 2 episodes
Aggregation — 2× in this paper
Optimal Transport — 2× in this paper · explained in 5 episodes
Sparsity — 2× in this paper · explained in 4 episodes
Cooperation — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Hyperparameters — 1× in this paper · explained in 2 episodes
Inverse Problem — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes

Summary

The gist A scalable method for approximating quadratic entropy using random feature embeddings and projections dramatically improves computational complexity for common distances like Euclidean and spherical geodesic cases, enabling computation at large scale through amortization techniques.

Negative Type Distances and Feature Embeddings

The paper considers distances of negative type, where a symmetric nonnegative matrix d in R n×n with zero diagonal satisfies dT da ≤ 0 for all a in R n with 1T a = 0. A classical result states that if d is negative type, there exist feature vectors ψ(j) in R n such that the distances are identically equal to the squared Euclidean distance between these features. This relationship is expressed as pT dp = 2 X j pj ψ(j) 2 − 2 X j pj ψ(j) 2.

Efficient Quadratic Entropy Computations for Common Distances of Negative Type

The paper details efficient estimators for common distances of negative type, focusing on squared Euclidean and cosine distances which yield trivially efficient computations. For ordinary Euclidean distance, an unbiased estimator Q¯euc N is computed by sampling IID vectors u ∼ N (0, Im) and computing the sample mean of q euc(u):= rπ/2 X j,k pjpkuT x(j) − uT x(k). This estimator is shown to be unbiased, with a relative error bound of P Q¯euc N / Q − 1 ≥ ε ≤ 2 exp(-Nε2/π).

Geodesic Distance on S m−1

For the spherical geodesic distance, the paper linearizes the geodesic distance through random sign features sj(u):= sgn(uT x(j)) to yield θjk = π/2 Eu(1 − sj sk). This leads to Theorem 3.7, which states that for d a geodesic distance matrix on S m−1 and u ∼ N (0, Im), pT dp = π/2 1 − Eu[s 2]. A corresponding estimator Q¯geo N is shown to be unbiased with a relative error bound of P Q¯geo N − Q ≥ ε ≤ 2 exp(-8Nε2/π2).

Scaling and Amortization

The paper addresses scaling by showing that both Euclidean and geodesic distance estimators can reuse a single product U T X across different p, provided only that X and d are preserved. This requires a single O(mnN) effort to form U T X followed by O(nN) for each Q. Furthermore, the Euclidean control variate w euc(u) is shown to be computable in O(mn), which is more efficient than the O(mnN) required for the sketch-based estimator.

Examples at Scale

The paper demonstrates these methods on real data from Open Graph Benchmark datasets, showing that feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. For instance, a bit-packed sign sketch requires roughly a thousand times less storage for N = 2500 compared to storing the n × n distance matrix, and the one-time sketch construction took just 3.76 seconds on a MacBook Pro. The results show that feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ.

Control Variates

A useful control variate for the Euclidean distance quadratic entropy is w euc(u):= X j,k pjpk(uT x(j) − uT x(k))2= 2X j pj (uT x(j)) squared − 2 uT X j pj x(j) squared. This control variate is computable in O(mn) versus O(mnN) for the sketch-based estimator. A control variate for the geodesic distance case is w geo(u):= (ˆuT P j pj x(j))2= 1/m P j pj x(j) squared. This is less effective for concentrated data than in the Euclidean case.

Conclusion

The paper proves that the sketch-based estimators provide unbiased estimates of quadratic entropy with error bounds independent of ambient dimension m. These methods allow for efficient computation and amortization over instances of p, making them suitable for large-scale applications in bibliometric and scientometric contexts. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ. The work provides a framework for approximating Q = pT dp for arbitrary distributions p and common distances d of negative type. The paper demonstrates that the feature-based quadratic entropy captures a ground-truth semantic dispersion of citing papers while controlling for the citation count χ.

Improvements for AI systems

The system can efficiently estimate quadratic entropy for millions of arbitrary distributions on a fixed set of points represented in Rm, enabling scalable analysis of citation dispersion or semantic spread from text features alone.
For Euclidean distance metrics, the AI can compute an unbiased estimator Q¯eucN using N IID vectors u ∼ N (0, Im) with a runtime of O(mnN), which is significantly faster than the naive O(mn 2) evaluation for large n.
For spherical geodesic distances, the AI can estimate quadratic entropy via a single scalar per random direction using the estimator Q¯geoN = 1/N X Ni=1 q geo(u(i)), achieving an unbiased estimate with a runtime of O(mnN).
The system can utilize control variates to minimize estimation variance, specifically employing the control variate w euc(u) = X j,k pjpk (u T x(j) - u T x(k)) squared for Euclidean cases to achieve a minimum variance of Var(Q¯eucN) = π 2/16Q squared.
The system can provide relative error guarantees for both distance types, such as PQ¯eucN/Q - 1 ≥ ε ≤ 2 exp(-Nε 2/π) for Euclidean entropy or PQ¯geoN - Q ≥ ε ≤ 2 exp(-8Nε 2/π2) for geodesic entropy.
The AI can perform large-scale bibliometric and scientometric analysis by stratifying citation counts (e.g., [10, 50], [50, 100]) to show how feature-based quadratic entropy captures a ground-truth (defined by arXiv subject area) semantic dispersion of citing papers.