Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning

arXiv:2610.07859 · cs.LG, cs.DC, cs.NI · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning".

Jane: The gist The proposed Tram-FL mechanism efficiently addresses computational and communication costs in decentralized federated learning by sequentially training a single model through model circulation among nodes How it works…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper now, "Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning." We've seen that it moves away from keeping multiple local models to circulating just one global model around the network sequentially.

Jane: That’s right. The naive question here is how do you get a decent global model when every single node is doing its own thing and constantly exchanging models? That sounds incredibly wasteful, especially with non-IID data where the data on each machine isn't the same as everyone else's.

The paper's summary: Tom: Moving past the mechanics, what does this actually achieve in terms of making DFL viable for real-world use? What’s the big picture here?

Jane: Well, the core idea is that by structuring the training through circulation, we can achieve a high accuracy global model even when you have non-IID data. The main takeaway is realizing decentralized federated learning with minimal computation and communication costs <ref:2610.07859#pg1>.

Lu: It’s about finding a way to manage the inherent challenges of non-IID settings so that we don't have to spend massive resources just to achieve decent accuracy <ref:2610.07859#pg3>. They propose a mechanism where each node directly updates the global model instead of relying on local models or complex aggregation methods <ref:2610.07859#pg2>.

Meng: So, it’s about finding a path that works well across different data distributions without requiring every single client to be perfectly synchronized at every step.

Lalam: It means we can achieve a high accuracy global model by carefully scheduling the circulation-based model training to mitigate the accuracy degradation caused by non-IID data <ref:2610.07859#pg2>.

Tom: Right, so it’s not just about getting an answer; it’s about making that answer achievable efficiently for everyone involved. The mechanism is designed to handle the training scheduling problem through model circulation <ref:2610.07859#pg1>.

Jane: And they achieve this by optimizing the route and update iterations based on those circulation dynamics <ref:2610.07859#pg1>. It’s a way to find a sweet spot between communication overhead and model accuracy when dealing with skewed data distributions <ref:2610.07859#pg2>.

Lu: They show that this sequential training approach is feasible for realizing DFL with minimal computation and communication costs, which is the whole point of the paper <ref:2610.07859#pg1>.

Meng: From a production perspective, if we can achieve good accuracy with less network traffic, that’s immediately better for running things on constrained devices or in slow environments.

Lalam: This approach allows us to get stable convergence even under non-IID conditions <ref:2610.07859#pg4>.

The paper's improvements: Tom: Let’s talk about the specific tweaks they introduced that make this better than what we currently have, specifically the mini-batch allocation and route determination.

Jane: I want to focus on how they handle the training schedule problem, because that seems like a major hurdle in DFL. They introduce a mini-batch number allocation step <ref:2610.07859#pg2>.

Lu: This step determines the number of training batches for every node so that their cumulative samples used for model training adhere to a pattern closely approximating a uniform distribution <ref:2610.07859#pg2>.

Meng: So, they're actively balancing the data usage across all nodes, aiming for unbiased learning based on all available labels <ref:2610.07859#pg2>. This sounds like a proactive defense against label skew.

Lalam: They formulate this selection policy as minimizing x sigma(L k + X i∈V Bxi Ni Li) to minimize the variance of L k, which suggests that the distribution of L k+one will be a uniform distribution <ref:2610.07859#pg2>.

Tom: That mathematical formulation is what makes it solid—it's not just a guess; it’s an optimization aimed at uniformity <ref:2610.07859#pg2>. How does that work when you have extreme label skew?

Jane: It works because even with extreme label skew, the mechanism compensates by allocating more batches to nodes that hold the scarce classes <ref:2610.07859#pg2>. That’s a really robust feature for real-world deployment.

Lu: Then we have the route determination step, which optimizes that circulation path <ref:2610.07859#pg1>. It orders nodes by their allocated batch counts in descending order of xi, keeping the last node of the previous cycle first <ref:2610.07859#pg3>.

Meng: So they prioritize nodes with more batches early on to avoid short-term training data bias <ref:2610.07859#pg3>. That feels like a very sensible heuristic for sequencing.

Tom: And the constraint they put on the last node continuing its training consecutively is what’s interesting—it’s expected to reduce communication frequency by one cycle per round <ref:2610.07859#pg3>.

Jane: That reduction in communication frequency is a direct benefit for deployment because fewer rounds mean less overhead overall <ref:2610.07859#pg3>.

Lu: And we also have the quantization compression during Step four where they use quantized momentum <ref:2610.07859#pg4>. This reduces the bit-width from thirty-two-bit floating point numbers to eight-bit or lower integer values <ref:2610.07859#pg4>.

Meng: That’s where we see the communication cost savings materialize directly—the paper shows that compression up to a certain threshold offers greater benefits in reducing communication traffic than the corresponding deterioration in accuracy <ref:2610.07859#pg4>.

Tom: So, they’ve got a multi-pronged approach: balancing batches for fairness, sequencing based on those batches for stability, and compressing the data being sent over to save bandwidth <ref:2610.07859#pg4>.

Jane: It sounds like a very well-thought-out system designed specifically to tackle the computational bottlenecks of decentralized learning.

Conclusion: Tom: So, wrapping up the Tram-FL paper, what’s the final word on its impact and where should we look next?

Jane: The implication is that efficient learning on non-IID datasets is feasible if you establish that optimal mini-batch number allocation and implement a route determination for choosing the most effective circulation route <ref:2610.07859#pg4>. It’s a concrete roadmap for better DFL.

Lu: They also pointed out their limitations, which are important to acknowledge—they mentioned targeting cross-silo FL, which limits parallelism <ref:2610.07859#pg4>. And they relied on pre-shared label histograms, and those privacy implications need more quantitative evaluation <ref:2610.07859#pg4>.

Meng: I agree with Lu on the reliability thing. If we can't guarantee reliable links or no node churn, that reliance on fixed distributions might break down under real operational stress <ref:2610.07859#pg4>.

Lalam: It seems like the system is very effective in terms of reducing communication and model updates compared to baseline methods like Gossip-SGD and PDMM-SGD <ref:2610.07859#pg4>.

Tom: Right, so we're looking at a novel DFL methodology based on sequential model circulation that incorporates compressed momentum <ref:2610.07859#pg1>. It achieves stable convergence with significantly reduced communication traffic and model updates for tasks like CIFAR-ten <ref:2610.07859#pg4>.

Jane: That’s the summary: a single global model shared optimally through scheduling and compression. It gives us a clearer path forward for making DFL practical even with messy, non-IID data <ref:2610.07859#pg1>.

Kota Maejima, Takayuki Nishio, Asato Yamazaki, Yuko Hara-Azumi

School of Engineering, Institute of Science Tokyo

cs.LG, cs.DC, cs.NI

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 12 pages, 7 figures, 5 tables. This work has been submitted to the IEEE for possible publication

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist The proposed Tram-FL mechanism efficiently addresses computational and communication costs in decentralized federated learning by sequentially training a single model through model

Key concepts

Model Circulation: This is the core idea where a single model is passed sequentially from one node to another in a predetermined route. Instead of independent training, nodes take turns updating the same global model, ensuring all nodes work toward a unified result without needing to maintain separate local copies.
Mini-batch Number Allocation: This step balances the amount of data each node trains on. It determines how many training batches each node receives so that the cumulative samples used across all nodes closely approximate a uniform distribution, which helps ensure unbiased learning from all available data classes.
Quantized Momentum Compression: To save communication bandwidth, Tram-FL reduces the size of model updates sent between nodes by quantizing momentum. This means converting high-precision floating-point numbers into smaller integer values, drastically lowering the data transmitted while maintaining acceptable model accuracy.
Route Determination: This step decides the order in which nodes receive and update the model during a cycle. The proposed method orders nodes based on their allocated batch counts to prevent short-term data biases and catastrophic forgetting by ensuring nodes trained earlier later receive updates.

Terminology

Decentralized Federated Learning — 1× in this paper · explained in 2 episodes
Federated Learning — 1× in this paper · explained in 17 episodes

Summary

The gist The proposed Tram-FL mechanism efficiently addresses computational and communication costs in decentralized federated learning by sequentially training a single model through model circulation among nodes

How it works

Tram-FL shifts the focus from conventional decentralized federated learning (DFL) which maintains multiple local models to circulating a single global model among nodes. This approach is designed to realize DFL with minimal computation and communication costs. The mechanism addresses the training scheduling problem by determining which nodes should update the model and the number of updates to perform through considering the model’s circulation route and update iteration allocation.

The process of Tram-FL involves four distinct steps in each cycle

Step 1 is the mini-batch number allocation step, which determines the number of training batches for every node to balance cumulative label usage.
Step 2 is the route determination step, which establishes the model’s circulation route for the cycle.
Step 3 is the local model and momentum update step, where each node updates the model parameters and momentum using mini-batches.
Step 4 is the model and momentum transmission step, where the model and quantized momentum are transmitted to the next node along the determined route.

Mini-batch Number Allocation

The mini-batch number allocation step determines the number of training batches for each node such that cumulative samples used for model training adhere to a pattern closely approximating a uniform distribution, specifically aiming to achieve l k c = l k c' (c, c' ∈ C). The selection policy is formulated as x∗ = argmin x σ(L k + X i∈V Bxi Ni Li). This optimization aims to minimize the variance of L k, which suggests that the distribution of L k+1 will be a uniform distribution, implying unbiased learning based on all available labels. This approach is effective even with extreme label skew where nodes without samples of a certain class are compensated by allocating more batches to nodes that do hold that class.

Route Determination

The route determination step optimizes the circulating route of the model to prevent short-term data biases and catastrophic forgetting. The proposed methodology orders nodes by their allocated batch counts in descending order of xi, keeping the last node of the previous cycle first. This ordering is anticipated to train nodes with a higher number of batches first, thereby avoiding short-term training data bias. Furthermore, this route determination is expected to prevent catastrophic forgetting by ensuring that nodes trained earlier are more likely to receive and train the model later. The constraint imposed is that the last node of a cycle continues training consecutively, which is expected to reduce communication frequency by one cycle per round.

Quantized Momentum Compression

To control communication load per transmission, Tram-FL incorporates quantized momentum during Step 4. Quantization reduces the bit-width of model parameters from 32-bit floating-point numbers to 8-bit or lower integer values. The conversion formula for quantization is given by vq = clip(round(s · v + z), −2b−1, 2b−1, −1). This technique is adopted because the precision of momentum does not significantly impact training, leading to a trade-off where communication traffic is reduced while model accuracy remains largely unaffected. Experimental results show that compression up to a certain threshold offers greater benefits in reducing communication traffic than the corresponding deterioration in accuracy.

Evaluation and Results

Experimental results demonstrate that Tram-FL achieved stable convergence with significantly reduced communication traffic and model updates compared to existing methodologies like Gossip-SGD and PDMM-SGD. For CIFAR-10, Tram-FL achieved a reduction of 83.6% in communication traffic and 96.1% in the number of model updates when reaching 40% accuracy. The results show that Tram-FL converged with less communication traffic than Gossip-SGD and PDMM-SGD across various scenarios, even when non-IID datasets are distributed among nodes. Moreover, the mini-batch number allocation makes the label usage exactly balanced over cycles, which can yield marginally better generalization on the classbalanced test set compared to centralized training.

Conclusion

In summary, Tram-FL proposes a novel DFL methodology based on model circulation that incorporates compressed momentum. It operates by sharing a single model with an optimal number of training batches and a designated circulation route to realize a global model. The study concludes that efficient learning on non-IID datasets is feasible if an optimal mini-batch number allocation is established, and a route determination is implemented for choosing the most effective circulation route. This suggests that efficient learning on non-IID datasets is feasible if an optimal mini-batch number allocation is established, and a route determination is implemented for choosing the most effective circulation route. The limitations noted include targeting crosssilo FL which limits parallelism, reliance on pre-shared label histograms whose privacy implications were discussed but not quantitatively evaluated, and assuming reliable links and no node churn. Future research will focus on adapting the proposed method to more realistic scenarios.

--- Page 1 ---

Evaluation and Results

Improvements for AI systems

Model circulation approach enables each node directly updates the global model by circulating it among nodes, which significantly reduces both computational and communication costs compared to conventional DFL methods that maintain multiple models in parallel.
Mini-batch number allocation is used to minimize distribution bias by determining the number of training batches for each node such that L k adheres to a pattern closely approximating a uniform distribution, aiming for unbiased learning based on all available labels.
Route determination optimizes the circulation path by ensuring that nodes with fewer training batches early in the cycle is anticipated to skew the data distribution used for model updates in the short term and by imposing a constraint where the last node of a cycle continues training consecutively, which is expected to reduce communication frequency by one cycle per round.
Quantized momentum compression reduces communication costs, as experiments showed that the 8-bit quantized momentum method achieved an average reduction of 40.09% in communication traffic compared to the non-quantized approach with negligible impact on accuracy.
The resulting system can achieve stable convergence with significantly reduced communication traffic and model updates, leading to a global model that can reach high accuracy thresholds (e.g., 80% or 90%) with substantially less communication overhead than baseline methods like Gossip-SGD or PDMM-SGD, as shown in Table V.

Abstract

Conventional decentralized federated learning (DFL) often focuses on clients, with each client maintaining a model copy, performing updates individually, and undertaking model exchange and integration. While fully leveraging computational resources can shorten training times, it can also lead to significant computational and communication waste. This is especially pronounced with non-independent and identically distributed (non-IID) data, where achieving high model accuracy demands extra resources. This research shifts focus to the model itself, aiming to realize DFL with minimal computation and communication costs. To this end, we propose Tram-FL (Traveling Model Training Mechanism for Decentralized Federated Learning), a mechanism designed to efficiently address these challenges. It sequentially trains a single model by circulating it among nodes. We address the training scheduling problem in model circulation-based training, specifically determining which nodes should update the model and the number of updates to perform. This is approached by considering the model's circulation route and update iteration allocation, for which we propose simple yet effective methods. Additionally, with quantized momentum, Tram-FL achieves high accuracy with fewer model circulations while controlling communication load per transmission. Experimental results show that the proposed algorithm, even with non-IID data, converges to a global model with reduced communication and computation.