Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

arXiv:2606.08252 · cs.CR, cs.DC · Submitted 2026-06-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning".

Elias: Here's the gist of the paper. The gist: This research presents a first theoretical and empirical analysis of a hidden privacy risk in logit-based FL methods, specifically that a semi-honest server may learn clients’ private models from logits during training.

Nadia: Quantifying the Privacy Risk The paper introduces the Adaptive Model Stealing Attack (AdaMSA) to quantify this risk by leveraging historical logits during training.

Elias: AdaMSA adaptively steals the private model by approximating its intermediate training states from prior iterations during training.

Nadia: The target logit is formulated as pˆT = ∑t=T−T0 t Z · p t, where T0 is the threshold governing the extent of past predictions to be considered, and Z = 1/∫T k=T − T0 k is a normalization factor that ensures the weights w t = 1/Z sum to 1.

Title and authors: Nadia: So we're looking at this paper called "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning." It sounds like they are tackling a specific weakness in how these methods share model outputs, which are called logits, instead of just sending model weights or gradients.

Elias: Exactly. The title tells us they have two main goals: first, to quantify this hidden privacy risk, and second, to develop a defense against it. It points out that logit-based FL methods are different because they share outputs on public data during training instead of raw data itself, and the paper argues that this difference introduces a risk.

Nadia: Right. So what's the core problem they're pointing out here? Is it just a general privacy issue, or is it tied specifically to these logit-based approaches?

Elias: It’s specific to semi-honest servers, which is an adversary who follows the protocol but tries to learn something private from the information they observe. The paper argues that this server can actually learn clients' private models just by looking at the logits being sent during training, which is a risk that has been overlooked.

Priya: From my side, I wonder what kind of information an adversary could actually learn from just the logits. Does it give them direct access to the private data, or is it something more abstract?

Nadia: It’s about approximating the private model itself through these transmitted logits. The paper introduces an attack called Adaptive Model Stealing Attack, or AdaMSA, to make this risk concrete and measurable.

Elias: AdaMSA works by using historical logits from previous training iterations to create a better target logit for the attacker's model. They formulate this target logit as pˆT equals sum t from T minus T0 t Z times p t, where T0 is a threshold governing how much history they look at.

Nadia: That sounds like they’re trying to build an attacker that can mimic the victim model by looking back through time during training. And what does this attack actually do in practice?

Elias: The goal of AdaMSA is to minimize the distance between the attacking model's output and a target logit, which they formulate as minimizing LCE between pˆT and p˜T using gradient descent to train an attacking model, denoted as ˜theta.

Priya: So, if I understand this correctly, they are saying that by looking at all those past logits—the historical ones—an attacker can build a very good approximation of the private model's state during training. That’s a lot of information to extract just from the outputs.

Nadia: Precisely. And then they quantify how bad this risk is by measuring the accuracy of that attacking model on some private test dataset, which they call R, where R is evaluated as E x∼Dk Acc f (x, ˜theta).

The paper's summary: Nadia: So we talked about what AdaMSA is. Now let’s look at the broader picture the authors are presenting in this paper. They lay out this framework to show that logit-based FL has a hidden privacy risk, and then they offer a concrete way to measure it and fix it.

Elias: The paper summarizes the situation by first setting up a theoretical analysis of the risk bound in logit-based FL, which involves looking at how empirical risks of mixed datasets relate to private datasets. They introduce Theorem three point four which bounds that difference between empirical risks by a term involving alpha and lambda, where lambda is defined as the sum of the empirical risks on both public and private datasets for a specific hypothesis h*.

Nadia: That seems like they’re trying to establish some mathematical boundaries on how much privacy you can expect when mixing public and private data in this context. So, what does that bound actually mean for anyone using these systems?

Elias: It means that the gap between the performance on a mixed dataset and the performance on the truly private dataset is controlled by a term (one minus alpha) times something involving D H, which is related to the difference in data distributions between public and private datasets <ref:2606.08252#pg1>. And this bound goes to zero when alpha approaches one.

Priya: That's interesting because it suggests that if you use more of the private data in your mix, the performance gap closes down mathematically. But what about the practical implication for a user who is just trying to keep their model safe?

Nadia: Well, practically speaking, this theoretical analysis gives them a way to understand how mixing data affects privacy guarantees under Rényi Differential Privacy. It shows that if you use an ensemble result computed with perturbed outputs, it stays (alpha, one/ki)-Rényi differentially private for any alpha greater than the maximum i alpha i <ref:2606.08252#pg1>.

Elias: And they link this to the sensitivity Delta squared of a function f. The composition property of Rényi Differential Privacy algorithms ensures that if you use perturbed outputs, the ensemble result remains (alpha, one/ki)-Rényi differentially private if gamma is greater than or equal to two times the natural log of one point two five divided by delta times Delta squared epsilon squared <ref:2606.08252#pg1>.

Priya: So they’re giving us a concrete condition on that perturbation scale gamma that needs to be met to maintain a certain level of privacy, and it depends on the sensitivity of the function f. That’s something you have to calculate beforehand.

Nadia: It sounds like they are moving from just identifying a risk, to providing the math needed to control it through data mixing and perturbation techniques. This sets up their next step, which is building a defense strategy.

The paper's improvements: Elias: Now they propose the defense mechanism called Federated Logit Perturbation, or FedLP. This strategy involves perturbing the transmitted logits of local models to defend against that model stealing risk we discussed earlier. They aim to prevent an adversary from replicating the functionality of the private model while keeping training performance high.

Nadia: The defense is framed as a constrained optimization problem where they want to maximize maxp′ G(thetaT−one p′) − G(thetaT−one p)a subject to the constraint that the distance between original and perturbed logits is bounded by some non-negative constant gamma <ref:2606.08252#pg1>.

Elias: The heuristic solver they suggest is quite simple: they just perturb the logit by setting p′ equals p plus Z times e j, where e j is a one-hot vector with a one in the j-th dimension of p and zeros everywhere else <ref:2606.08252#pg1>.

Nadia: That sounds like a very specific, localized change to the logit—basically picking one dimension and adding a small value based on some normalization factor Z. Is that how simple it is for an attacker to bypass this?

Elias: The paper shows that this perturbation based defense strategy preserves (epsilon, delta)-DP if and only if gamma is greater than or equal to two times the natural log of one point two five divided by delta times Delta squared epsilon squared <ref:2606.08252#pg1>.

Priya: So the defense depends on a specific mathematical relationship between the noise level you introduce, which is controlled by gamma, and how sensitive the function f is. If you choose a small enough gamma that satisfies that inequality, you get your privacy guarantee.

Nadia: It sounds like they are providing a way to tune this trade-off between utility and privacy directly by setting that specific threshold for gamma based on the desired epsilon and delta values. That’s useful because it moves beyond just applying generic noise.

Elias: And they mention that the composition property of Rényi Differential Privacy algorithms means you can use these perturbed outputs in an ensemble calculation, and the resulting result stays (alpha, one/ki)-Rényi differentially private for any alpha greater than the maximum i alpha i <ref:2606.08252#pg1>.

Priya: So what we have here is a mechanism where we take a set of perturbed logits and combine them, and as long as the perturbation scale gamma is set correctly according to those parameters, the final result maintains that privacy level. That’s a useful way to think about maintaining privacy across multiple clients.

Conclusion: Nadia: So we've gone through AdaMSA and FedLP. To wrap up this paper "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning," the main point is that logit-based FL methods have a hidden privacy risk where a semi-honest server can learn private models from logits.

Elias: And they gave us tools to measure this risk with AdaMSA and then a concrete defense, FedLP, which uses logit perturbation to keep the utility high while bounding the perturbation scale by gamma.

Nadia: They showed that you can tune that gamma based on your desired privacy parameters epsilon and delta, providing a clear way to balance utility and privacy in these systems. This is important because it addresses a vulnerability specific to models that share logits during training.

Priya: I just want to say that the paper shows how important it is to be precise about what the data actually represents before trusting the claimed results, especially when dealing with these privacy guarantees under Rényi Differential Privacy, where you have to be careful with those parameters.

Elias: It’s a lot of work establishing these theoretical bounds, but they provide a framework for anyone working in this area who is looking to understand how to actually control the distance between private and public data using mixed datasets.

Nadia: This work on "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning" gives us a solid starting point for understanding these new privacy vulnerabilities that are specific to logit-based FL.

HKUST & SUSTech

cs.CR, cs.DC

Submitted: 2026-06-06

Updated: 2026-10-21

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist: This research presents a first theoretical and empirical analysis of a hidden privacy risk in logit-based FL methods, specifically that a semi-honest server may learn clients’ private

Key concepts

Adaptive Model Stealing Attack (AdaMSA): This attack quantifies the privacy risk by using historical logits from training iterations. It adaptively steals the private model by approximating its intermediate training states from past predictions, allowing an attacker to steal any desired private model during training.
Federated Logit Perturbation (FedLP): A defense strategy that perturbs transmitted logits to prevent adversaries from replicating the private model. It optimizes the perturbation scale ($\gamma$) to maximize performance while ensuring a strong privacy guarantee against model inference.
Empirical Risk Bound: This theoretical analysis establishes mathematical limits on how much the empirical risk of a mixed dataset can differ from the true private dataset risk, controlled by a parameter $\alpha$. This helps researchers quantify the trade-off between using public and private data in FL.
Rényi Differential Privacy (RDP): A formal privacy guarantee used to measure how much information an algorithm leaks about its input data. The paper shows that the FedLP defense preserves $(\epsilon, \delta)$-RDP properties for the ensemble result under certain conditions.

Terminology

Federated Learning — 2× in this paper · explained in 17 episodes
Communication Overhead — 1× in this paper · explained in 2 episodes

Summary

The gist: This research presents a first theoretical and empirical analysis of a hidden privacy risk in logit-based FL methods, specifically that a semi-honest server may learn clients’ private models from logits during training.

Quantifying the Privacy Risk

The paper introduces the Adaptive Model Stealing Attack (AdaMSA) to quantify this risk by leveraging historical logits during training. AdaMSA adaptively steals the private model by approximating its intermediate training states from prior iterations during training. The target logit is formulated as pˆT = ∑t=T−T0 t Z · p t, where T0 is the threshold governing the extent of past predictions to be considered, and Z = 1/∫T k=T − T0 k is a normalization factor that ensures the weights w t = 1/Z sum to 1. The attack then minimizes LCE (pˆT, p˜T) using gradient descent to train an attacking model ˜theta. The privacy risk R is quantified by evaluating the accuracy of the obtained attacking model f (x, ˜theta) on the victim’s private test dataset Dk as R = E x∼Dk Acc f (x, ˜theta).

Theoretical Analysis of Risk Bound

The theoretical analysis establishes bounds for this privacy risk in logit-based FL. The empirical risk of the mixed dataset Dmix is expressed as epsilonDmix (theta) = alphaepsilonDpr iv (theta) + (1 − alpha)epsilonDpub (theta), where Dmix = (S1, S2), with S1 consisting of αDmix instances sampled independently from Dpr iv and S2 consists of (1 − α)Dmix instances sampled independently from Dpub. Theorem 3.4 bounds the difference between the empirical risks of the mixed dataset and the private dataset by epsilonDmix (h) − epsilonDpr iv (h) ≤ (1 − α) (1/2 dHΔH (Dpr iv, Dpub) + λ), where λ = epsilonDpr iv (h∗) + epsilonDpub (h∗). This bound drops to 0 as α increases to 1.

Defense Strategy: Federated Logit Perturbation (FedLP)

The paper proposes a defense strategy named Federated Logit Perturbation (FedLP) that perturbs the transmitted logits of local models to defend against this privacy risk. The defender aims to prevent an adversary from being able to replicate the functionality of its private model while preserving training performance by bounding the perturbation scale by a non-negative constant gamma: p − p′b ≤ gamma, where p is the original logit on the public data, p′ is the corresponding perturbed logit. The defense problem is reformulated as a constrained optimization problem to maximize maxp′ G(thetaT−1, p′) − G(thetaT−1, p)a (10) subject to p − p′b ≤ gamma. A simple heuristic solver perturbs the logit by setting p′ = p + Z · e j, where e j denotes a one-hot vector with 1 in the j-th dimension of p and 0’s elsewhere.

Privacy Guarantee of FedLP

The proposed perturbation based defense strategy in Algorithm 2 preserves (epsilon, delta) − D P if and only if gamma ≥ 2ln(1.25/δ · (Δ2 ε)) squared. The mechanism described in Algorithm 2 with a sensitivity Δ2 preserves (epsilon, delta) − D P if and only if gamma ≥ 2ln(1.25/δ · (Δ2 ε))2, where Δ2 (f) is the l2-sensitivity of a function f. The composition property of RDP algorithms ensures that the ensemble result computed using the perturbed outputs remains (alpha, Ík i=1 delta)-Rényi differentially private for any alpha ≥ max i α i.

Experimental Validation

The paper empirically validates AdaMSA and FedLP in three experimental settings: Close-world, Open-world-CF, and Open-world-TI. The results show that AdaMSA demonstrates significant improvements of up to 4.00%, 3.41%, and 3.11% in the respective settings compared to MSA [36] and Naïve AdaMSA baselines. Furthermore, our proposed FedLP defense is closest to the ideal trade-off curve, indicating it achieves a better utility and privacy trade-off compared to state-of-the-art baselines. The results confirm that increasing the value of α (i.e., decrease the distance), will increase the utility at the cost of increasing privacy risk. Finally, we observe that when T0 increases from 0 to 3, the attack performance gradually increases by 2.87% in Open-world-CF setting and 2.45% in Open-world-TI setting. The results also show that AdaMSA can achieve similar accuracy of the victim models under different model hyperparameters.

Conclusion

The paper provides the first theoretical and empirical analysis of a hidden privacy risk in logit-based FL that the semi-honest server can infer clients’ private models according to logits. The research introduces AdaMSA to quantify this risk and FedLP as a simple but effective perturbation-based defense strategy. The experimental results showcase the effectiveness of AdaMSA and FedLP in various scenarios. This work highlights the importance of addressing privacy vulnerabilities in logit-based FL methods.

REFERENCES

[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 308–318

[2] Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 (2020)

[3] Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. In The Eleventh International Conference on Learning Representations

[4] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. 2007. Learning bounds for domain adaptation. NeurIPS 20 (2007)

[5] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In IJCNN. IEEE, 2921–2926

[6] Koby Crammer, Michael Kearns, and Jennifer Wortman. 2008. Learning from Multiple Sources. JMLR 9, 8 (2008)

[7] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference. Springer, 265–284

[8] Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9, 3–4 (2014), 211–407

[9] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting gradients-how easy is it to break privacy in federated learning? NeurIPS 33 (2020), 16937–16947

[10] Xuan Gong et al. 2021. Ensemble attention distillation for privacy-preserving federated learning. In ICCV. 15076–15086

[11] Xuan Gong et al. 2022. Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In AAAI, Vol.

Improvements for AI systems

Adaptive Model Stealing Attack (AdaMSA) can be used to systematically approximate a client's private model by leveraging historical logits during training to generate a more informative target logit for the attacking model. This allows an adversary to steal any desired intermediate private model throughout training.
Federated Logit Perturbation (FedLP) can be implemented as a defense strategy that perturbs transmitted logits in the direction that optimally obstructs the adversary while maximally preserving the model performance. This ensures a better trade-off between utility and privacy, achieving results closest to the ideal trade-off compared to state-of-the-art defenses.
The proposed system can be used in Open-world settings where public data is unrelated to private data, as the analysis shows that this inherent privacy risk persists even when public data is unrelated to private data, and the defense strategy maintains utility.
By using a mixed dataset construction with a weighting parameter alpha, the system can be tailored to control the distance between private and public datasets, allowing researchers to quantify and control the distance between the mix and private datasets.
The defense mechanism can be calibrated by setting Gaussian noise variance sigma based on the perturbation constraint scale gamma such that each individual query preserves (alpha, epsilon)-RDP, ensuring strong privacy guarantees under Rényi Differential Privacy.