Over-the-Air Ensemble Inference with Model Privacy

arXiv:2202.03129 · cs.LG, cs.CR, eess.SP · Submitted 2022-02-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Over-the-Air Ensemble Inference with Model Privacy".

Jane: Here's the gist of the paper. The gist: This work introduces an over-the-air ensemble inference framework that exploits the superposition property of air to enable bandwidth-efficient and private distributed edge inference using multiple independently trained models.

Tom: System Model and Problem Definition The system model considers privacy-preserving ensemble classification at the wireless edge, where there are n clients each with a separate trained model fi: R d → R k, i ∈

n: , for a classification task<ref:2202.03129#pg6>.

Jane: The clients are connected to a central inference server (CIS) via a wireless medium, and at time t, each client i knows its channel gain hi,t ∈ C<ref:2202.03129#pg6>.

Tom: To reduce total power consumption and amplify privacy guarantees, the work considers random participation of the clients in each inference round such that each client i independently participates with probability p<ref:2202.03129#pg6>.

Title and authors: Jane: So we’re talking about "Over-the-Air Ensemble Inference with Model Privacy" by Selim F. Yilmaz, Burak Hasırcıoglu, and Deniz G ˘ und ¨ uz¨ This title tells us immediately that the focus is on putting models in the air to make a decision without sending everything back.

Tom: Right. The authors are really pushing for two things here: maximizing inference accuracy while simultaneously maximizing privacy for those local models. They want to do both at the edge, using an ensemble of independently trained models, and they're focusing on how to do that efficiently over a wireless link.

Lu: It sets up the core problem as an ensemble inference task where individual hypotheses from nodes need to be conveyed and combined for the best decision, which is a standard way to think about distributed learning in this context <ref:2202.03129#pg2>.

Meng: But how do you actually do that without drowning everyone in communication overhead? That’s usually where I run into trouble when deploying these kinds of systems at scale.

Jane: Well, this paper proposes different over-the-air ensemble methods specifically to address that bandwidth issue, showing they can perform significantly better than the standard orthogonal methods in terms of resources used <ref:2202.03129#pg1>.

Tom: They show that these schemes use less resource and provide privacy guarantees while still performing well on the task at hand. It’s about finding a sweet spot between speed, accuracy, and privacy protection.

Lu: And they explore two different local prediction methods: one is a belief summation method, where you sum up the beliefs of all participating clients and then the server picks the highest total score <ref:2202.03129#pg2>.

Meng: The second method sounds more complex, like a majority voting with an over-the-air computation method where clients vote and then the server picks the class with the most votes <ref:2202.03129#pg2>.

Jane: And crucially, they show how to embed privacy mechanisms directly into this inference phase instead of just training the models in a differentially private way, which is a key difference from other work in this area <ref:2202.03129#pg3>.

The paper's summary: Tom: So summarizing what we’ve covered, this paper is about enabling distributed inference at the edge using an ensemble of models trained separately on local data, and they focus on making it bandwidth-efficient by exploiting the air.

Jane: They introduce two different ways to make those local predictions—a belief summation method and a majority voting with over-the-air computation—and they show these methods beat standard orthogonal ones in terms of resource use.

Lu: The paper focuses heavily on embedding privacy mechanisms into the inference phase because training models in a differentially private way doesn't stop model stealing attacks, which is important context <ref:2202.03129#pg3>.

Meng: So the main contribution seems to be their specific approach to noise addition—they add Gaussian noise to the prediction, g i(x t) = f i(x t) + m i,t, where m i,t is normally distributed <ref:2202.03129#pg3>.

Tom: And they show that the privacy guarantees directly depend on the variance of this aggregated noise at the central inference server. They even have a trick to keep those DP guarantees independent of how many clients are participating <ref:2202.03129#pg3>.

Jane: That trick involves having each client add noise with a variance scaled by one over the number of participating clients, P t, so sigma two client = sigma squared / P t <ref:2202.03129#pg3> <ref:2202.03129#pg2>.

Lu: This allows them to achieve DP guarantees that depend on the desired privacy parameters and not on the size of the participating set P t <ref:2202.03129#pg3>.

The paper's improvements: Tom: Now, what are the actual improvements they claim? They focus on how their proposed over-the-air ensemble methods compare to their standard orthogonal counterparts in terms of performance and resource usage.

Jane: The main finding is that the OAC-based methods perform significantly better than their orthogonal counterparts when privacy is involved, and they do this while using less resources <ref:2202.03129#pg1>.

Lu: They also show that in the non-private setting, when you look at the best client model and orthogonal methods, those perform near-random results <ref:2202.03129#pg1>.

Meng: So what does that mean for deployment? It suggests that if you need privacy, sticking with these OAC methods gives you a much better accuracy trade-off than just using standard ensemble techniques.

Tom: And they also found that higher participation probability p improves the performance significantly in the private setting when the privacy parameter epsilon is set to one <ref:2202.03129#pg1>.

Jane: They also observed something interesting about transmitting class scores; they found that transmitting those scores from each client makes an ensemble method more informative, which can make local decisions more reliable when noise is introduced for privacy <ref:2202.03129#pg1>.

Lu: So the improvements boil down to better resource efficiency and stronger privacy guarantees compared to existing methods, especially in a distributed inference setting <ref:2202.03129#pg1>.

Conclusion: Tom: So wrapping up this discussion on "Over-the-Air Ensemble Inference with Model Privacy," the core idea is that you can get private, bandwidth-efficient ensemble inference at the edge by exploiting the air and using specific ensemble methods.

Jane: The main implication for listeners is that this framework offers a way to achieve differential privacy guarantees without having to rely on training data being private, which is a big step forward in how we think about edge AI <ref:2202.03129#pg3>.

Lu: It shows that embedding the noise mechanism directly into the inference phase, combined with random client participation, allows for flexible privacy control depending on the scenario <ref:2202.03129#pg1>.

Meng: From an engineering standpoint, it’s clear that this approach is much better than orthogonal methods if your main goal is balancing accuracy against communication costs when privacy is a requirement <ref:2202.03129#pg1>.

Lalam: I confirm that the paper establishes (ε, δ)-DP guarantees for both the case where all clients participate and the case where they participate with probability p < one <ref:2202.03129#pg4> <ref:2202.03129#pg1>.

Tom: Yeah, so it's about getting reliable results at the edge without sending too much data back to a central server <ref:2202.03129#pg1>.

Jane: It’s a really solid piece of work because it provides concrete methods and analysis for handling privacy in this distributed environment <ref:2202.03129#pg3>.

Selim F. Yilmaz, Burak Hasırcıoglu, Deniz G ı und ı uz࡯

Imperial College London · University of Modena and Reggio Emilia

cs.LG, cs.CR, eess.SP

Submitted: 2022-02-07

Updated: 2022-05-15

Comments: To appear in IEEE International Symposium on Information Theory (ISIT) 2022

Journal ref: Proc. IEEE ISIT 2022, pp. 1265-1270

DOI: 10.1109/ISIT50566.2022.9834591

Code: https://github.com/selimfirat/oac-based-private-ensembles

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: This work introduces an over-the-air ensemble inference framework that exploits the superposition property of air to enable bandwidth-efficient and private distributed edge inference using

Key concepts

Over-the-Air Ensemble Inference: This method involves using several different machine learning models trained separately on local data at various clients. Instead of sending all predictions, the system combines these multiple model outputs to make a final decision at a central server, improving accuracy while keeping communication efficient.
Random Participation: To save power and enhance privacy, not every client participates in every inference round. Clients only join the process if their channel gain is above a certain threshold. This random selection limits the number of participants, which helps control noise variance and ensures that privacy guarantees hold even when clients are randomly selected.
Embedding Privacy Mechanisms: Privacy is achieved by adding controlled Gaussian noise directly to each client's prediction before transmission, rather than just training models with differential privacy. This technique allows the system to maintain strong privacy guarantees regardless of how many clients participate in a given round.
Channel Inversion: Clients scale their noisy scores by the inverse of their channel gain ($1/h_{i,t}$). This step effectively cancels out the effect of the wireless channel on the prediction, allowing the central server to receive a signal that is easier to interpret and recover a reliable decision from.

Terminology

Accuracy — 1× in this paper · explained in 1 episode

Summary

The gist: This work introduces an over-the-air ensemble inference framework that exploits the superposition property of air to enable bandwidth-efficient and private distributed edge inference using multiple independently trained models.

System Model and Problem Definition

The system model considers privacy-preserving ensemble classification at the wireless edge, where there are n clients each with a separate trained model fi: R d → R k, i ∈ [n], for a classification task<ref:2202.03129#pg6>. The clients are connected to a central inference server (CIS) via a wireless medium, and at time t, each client i knows its channel gain hi,t ∈ C<ref:2202.03129#pg6>. To reduce total power consumption and amplify privacy guarantees, the work considers random participation of the clients in each inference round such that each client i independently participates with probability p<ref:2202.03129#pg6>. Only clients whose channel gains are larger than a certain threshold participate the inference, which determines p<ref:2202.03129#pg6>. Each participating client makes a prediction denoted by fi(xt) and has a bandwidth of k channel uses to convey their predictions to the CIS.

Ensemble Methods

The framework introduces two different ensemble methods for making local predictions based on the received query xt.

Belief summation method sums beliefs of the participating clients for all the classes and the CIS later selects the class with the highest total score, using model fi(xt) = ri,t.
Majority voting with OAC method allows participating clients to vote for a class and the CIS later selects the class with highest number of votes, using model fi(xt) = ToOneHot arg max j∈[k] (ri,t)j, k!

Ensuring Privacy

Privacy is ensured by embedding mechanisms into the inference phase rather than relying solely on differentially private training. The work focuses on embedding privacy-preserving mechanisms into the inference phase because DP training does not prevent model stealing attacks. A Gaussian noise is added to the model prediction, resulting in gi(xt) = fi(xt) + mi,t, where mi,t ∼ N (0, σ squared clientIk). The analysis reveals that DP guarantees are directly dependent on the variance of the aggregated noise at the CIS. To obtain DP guarantees independent of the number of participating clients, each client should add a Gaussian noise with σ squared client = σ 2/Pt, where σ squared is a constant depending on the desired DP guarantees and Pt is the set of participating clients.

Transmission and Final Decision

Each client scales its noisy score by 1/hi,t using channel inversion to cancel its effect. The transmitted signal yi,t is defined as yi,t = Atgi(xt)/hi,t for i ∈ Pt and otherwise (6)<ref:2202.03129#pg9>. The received signal at the CIS is zt = At X i∈Pt mi,t + X i∈Pt fi(xt)! + nt. The variance of total noise received by the CIS at time t is σ squared CIS = σ squared channel + PtA squared t σ squared client, i.e., zt ∼ N (0, σ2 CISIk). After receiving zt, the CIS multiplies the received signal by 1/At to recover the decision.

Privacy Analysis

Theorem 1 states that if all clients participate in inference, i.e., p = 1, then Algorithm 1 is (ε, δ)-DP such that for any ε > 0, δ = Φ(1/(√2σ) − εσ/√2) − e εΦ(−1/(√2σ) − εσ/√2). Theorem 2 shows that if each client independently participates in inference with probability p < 1, then Algorithm 1 is (ε0, δ0)-DP.

Simulations and Comparison

The experiments employ four different datasets: CIFAR10, CIFAR-100, FashionMNIST and IMDB. The results compare the proposed OAC-based methods with the best client model and ensemble methods with orthogonal transmission. The simulations show that OAC-based methods outperform orthogonal ones in the private setting when privacy is involved. Furthermore, OAC-based methods perform competitively in the non-private setting, when privacy is involved, best client model and orthogonal methods perform near-random, and significantly worse than the OAC-based methods. In Fig. 2, it is observed that higher p improves the performance significantly in the private setting (ε = 1). The paper concludes that distributed edge inference with OAC performs significantly better than its orthogonal counterpart while using less resources.

Conclusion

The framework has introduced a private edge inference framework with ensembling, exploiting OAC for bandwidth-efficient and private wireless edge inference for the first time in the literature. The work provides DP guarantees exploiting both distributed noise addition and random participation. The proposed framework performs significantly better than orthogonal counterparts while using less resources. The authors also observe that transmitting class scores from each client is more informative as an ensembling method, making and transmitting local decisions can be more reliable when noise is introduced to guarantee privacy.

How it works

The system model considers privacy-preserving ensemble classification at the wireless edge, where there are n clients each with a separate trained model fi: R d → R k, i ∈ [n], for a classification task<ref:2202.03129#pg6>. The clients are connected to a central inference server (CIS) via a wireless medium, and at time t, each client i knows its channel gain hi,t ∈ C. To reduce total power consumption and to amplify privacy guarantees, the work considers random participation of the clients in each inference round such that each client i independently participates with probability p<ref:2202.03129#pg6>. Only clients whose channel gains are larger than a certain threshold participate the inference, which determines p. Each participating client makes a prediction denoted by fi(xt) and has a bandwidth of k channel uses to convey their predictions to the CIS.

Ensuring Privacy

Privacy is ensured by embedding mechanisms into the inference phase rather than relying solely on differentially private training. A Gaussian noise is added to the model prediction, resulting in gi(xt) = fi(xt) + mi,t, where mi,t ∼ N (0, σ squared clientIk). The analysis reveals that DP guarantees are directly dependent on the variance of the aggregated noise at the CIS. To obtain DP guarantees independent of the number of participating clients, each client should add a Gaussian noise with σ squared client = σ 2/Pt, where σ squared is a constant depending on the desired DP guarantees and Pt is the set of participating clients.

Privacy Analysis

Theorem 1 states that if all clients participate in inference, i.e.

Improvements for AI systems

Improved AI system can perform private distributed inference at wireless edge nodes by exploiting over-the-air computation (OAC) to achieve bandwidth efficiency while providing differential privacy guarantees. This allows clients to make predictions without revealing sensitive information about their local training datasets to the central inference server.
The system can offer flexible privacy guarantees depending on the scenario without imposing restrictions on the training phase, which is achieved by embedding privacy-preserving mechanisms directly into the inference phase rather than relying solely on differentially private training.
The system can be optimized for resource constraints by using random participation of clients in each inference round, where only the clients whose channel gains are larger than a certain threshold participate the inference, thereby limiting power consumption.
The system can compare and discuss privacy of introduced ensemble methods, showing that the proposed framework with OAC performs significantly better than orthogonal counterparts while using less resources. This means the system achieves higher accuracy and lower communication overhead compared to traditional ensemble methods when privacy is enforced.