Bilateral Sharpness-Aware Minimization for Flatter Minima

arXiv:2409.13173 · cs.CV, cs.LG · Submitted 2024-09-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Bilateral Sharpness-Aware Minimization for Flatter Minima".

Jane: Here's the gist of the paper. The gist The proposed Bilateral Sharpness-Aware Minimization (BSAM) method enhances generalization performance by combining Max-Sharpness (MaxS) and Min-Sharpness (MinS) to find flatter minima than standard Sharpness-Aware Minimization (SAM).

Tom: Sharpness-Aware Minimization (SAM) is an optimization technique that seeks a flat minimum associated with improved generalization performance by minimizing both the loss value and the loss sharpness.

Jane: The optimization objective of SAM is defined as min w max ε≤ρ L(w + ε) − L(w) + L(w) + λw2 squared, (1).

Tom: The sharpness in SAM is defined as h max(w) = max ε≤ρ L(w + ε) − L(w), which captures the sharpness of the loss by measuring how quickly the training loss can be decreased by moving from w to a nearby parameter value.

Paper summary: Tom: So we're looking at this paper called "Bilateral Sharpness-Aware Minimization for Flatter Minima." It talks about improving generalization by finding flatter minima than what standard Sharpness-Aware Minimization, or SAM, finds.

Jane: That sounds like a fancy way of saying they want models that are more robust when the data changes a little bit. What's the core idea here?

Tom: The paper starts by saying that SAM uses Max-Sharpness, or MaxS, to find flatter minima and improve generalization. But they found a problem with that approach, which is what they call the Flatness Indicator Problem, or FIP.

Lu: Exactly. The FIP is that SAM only looks at flatness in the direction of gradient ascent when it's trying to minimize loss >

Jane: So, if it only looks in one direction, it might not find a really flat region for the model to settle into. What does this mean for us?

Tom: It means the next spot SAM suggests might not be as flat as we hope. And because SAM is a greedy search method, they felt we needed a better way to measure flatness.

Meng: From an engineering standpoint, if you're doing this on production models, you need confidence that the optimization process is actually converging to a good solution, not just getting stuck in a shallow local minimum.

Jane: So how do they fix this flatness issue? What new idea are they proposing to make the indicator better?

Tom: They introduce Min-Sharpness, or MinS. This measures something different—the difference between the training loss and the minimum loss over a neighborhood around the current weight. It tells you how fast the loss decreases in both directions.

Lu: That’s interesting because they propose merging MaxS and MinS into something new, called Bilateral Sharpness, or BilS. This new indicator should show a flatter direction during optimization >

Jane: So this Bilateral Sharpness, this combination of MaxS and MinS, is supposed to give them a better guide than just the old MaxS when searching for minima >

Tom: Right. They say BSAM uses this combined indicator to find a more flat minimum than SAM does. They even have a formula for the gradient update in BSAM that tries to resolve a conflict between finding the training loss and finding these flatter regions >

Meng: That sounds complex for deployment. If you're running this on millions of parameters, how much overhead is there for calculating that MinS and then balancing it against MaxS? What breaks at scale?

Lalam: Based on the literature, BSAM aims to find a loss landscape after training with BSAM that shows both MaxS and MinS combined >

Jane: But what about the numbers? What did they actually show in their experiments to prove this method works better than vanilla SAM or even SGD?

Tom: They did extensive experiments across classification, transfer learning, human pose estimation, and network quantization. On image classification with ResNet-eighteen on CIFAR-one hundred trained for two hundred epochs, BSAM achieved an accuracy of eighty-five point five one plus or minus zero point fourteen >

Lu: That's better than SAM's accuracy of eighty-four point seven two plus or minus zero point eleven on the same task, which is a tangible improvement >

Jane: And they also showed that BSAM finds a smaller maximum eigenvalue on the test set compared to SGD and SAM, which is another way to show it finds flatter minima >

Tom: They also demonstrated robustness against label noise; models trained with BSAM maintained over seventy percent accuracy even when the noise rate reached eighty percent for CIFAR-ten and CIFAR-one hundred >

Meng: Robustness to noise is good, but I worry about generalization on new, unseen data. Does this flatness improvement translate reliably outside of these specific tasks?

Lalam: The theoretical analysis proves that BSAM converges to local minima >

Jane: So the authors are confident that this method consistently finds a good minimum across different types of tasks, not just those they tested on >

Tom: They also showed that for 2D Human Pose Estimation, BSAM performed better on key body joints like the Elbow, Wrist and Hip > <ref:2409.13173#pg2>

Jane: It sounds like the Bilateral SAM method is designed to be more thorough in exploring the parameter space than previous methods because it looks at both ascent and descent directions simultaneously >

Tom: So to wrap up this paper, they propose Bilateral SAM to find flatter minima by combining MaxS and MinS, and their experiments show it's superior to vanilla SAM across several tasks >

Jane: And the authors conclude that BSAM outperforms SGD and SAM on various test datasets, suggesting it genuinely improves model generalization >

Tom: It seems like a solid piece of work for anyone trying to push the limits of what optimization techniques can do for deep learning models >

Conclusion: Tom: So we're wrapping up on this paper about Bilateral Sharpness-Aware Minimization for Flatter Minima by Foret, Kleiner, Mobahi, and Neyshabur.

Jane: It basically boils down to taking the standard Sharpness-Aware Minimization SAM and making it smarter by looking at both how fast the loss goes down in one direction and how fast it goes down in another direction.

Lu: The authors found that SAM’s way of measuring flatness, MaxS, only looked in one direction during optimization, which they called the Flatness Indicator Problem.

Meng: Which means the next parameter point SAM suggests might not be as flat as we need it to be for good generalization when you deploy it on real traffic.

Lalam: They fixed this by introducing Min-Sharpness, MinS, which measures how quickly the training loss drops over a neighborhood around the current weight.

Tom: And they merged that with MaxS to create this Bilateral Sharpness indicator, BilS, which they claim gives a much better picture of flatness during optimization.

Jane: So what does this actually mean for a developer or someone using deep learning models? It means we’re trying to find parameter settings that are more stable and less sensitive to small changes in the input data.

Lu: The authors showed through experiments on classification, transfer learning, and even human pose estimation that BSAM finds these flatter minima consistently across different types of tasks.

Meng: But what about the practical side? The paper shows it converges to local minima, which is good for theory, but does that translate to a stable training run without needing super long training times?

Tom: They did show convergence bounds, specifically Theorem one which gives us some mathematical proof about how fast BSAM settles into a minimum.

Jane: It’s about making the model more robust when it sees data it hasn't seen before, which is the main goal of generalization in AI.

Lalam: BSAM consistently outperforms vanilla SAM and even standard SGD on various test sets, proving that this approach actually helps models generalize better across different scenarios.

Tom: So for anyone building a model where generalization matters most, this suggests we can get a more reliable result by looking at the whole landscape instead of just one side of it.

Jane: It opens up a new way to optimize neural networks by considering multiple aspects of the loss surface simultaneously, and we need to see if this idea pops up in other areas soon.

Jiaxin Deng, Junbiao Pang, Baochang Zhang, Qingming Huang

Beijing University of Technology · Beihang University · University of Chinese Academy of Sciences

cs.CV, cs.LG

Submitted: 2024-09-20

Updated: 2026-10-21

Code: https://github.com/ajiaaa/BSAM

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The proposed Bilateral Sharpness-Aware Minimization (BSAM) method enhances generalization performance by combining Max-Sharpness (MaxS) and Min-Sharpness (MinS) to find flatter minima than

Key concepts

Max-Sharpness (MaxS): This is the sharpness metric used in standard SAM. It measures how quickly the training loss can decrease when moving from the current weight to a nearby point in the direction of gradient ascent. It focuses on finding a minimum that is flat along this specific ascent path.
Flatness Indicator Problem (FIP): This problem occurs because MaxS only considers flatness in one direction—the gradient ascent direction. This leads to a next minimization region that is not sufficiently flat, limiting the generalization benefits of standard SAM.
Min-Sharpness (MinS): MinS addresses the FIP by measuring how quickly the training loss decreases when moving from the current weight to a nearby point in any direction. It captures flatness by considering both ascent and descent directions, providing a more balanced view of the parameter space.

Terminology

Generalization — 4× in this paper · explained in 13 episodes
Flatness Indicator — 2× in this paper
Sharpness-Aware Minimization — 2× in this paper
Quantization — 1× in this paper · explained in 15 episodes
Robustness — 1× in this paper · explained in 6 episodes
Sharpness-Aware Minimization (SAM) — 1× in this paper · explained in 3 episodes
Transfer Learning — 1× in this paper · explained in 15 episodes

Summary

The gist The proposed Bilateral Sharpness-Aware Minimization (BSAM) method enhances generalization performance by combining Max-Sharpness (MaxS) and Min-Sharpness (MinS) to find flatter minima than standard Sharpness-Aware Minimization (SAM).

Background of SAM

Sharpness-Aware Minimization (SAM) is an optimization technique that seeks a flat minimum associated with improved generalization performance by minimizing both the loss value and the loss sharpness. The optimization objective of SAM is defined as min w max ε≤ρ L(w + ε) − L(w) + L(w) + λw2 squared, (1). The sharpness in SAM is defined as h max(w) = max ε≤ρ L(w + ε) − L(w), which captures the sharpness of the loss by measuring how quickly the training loss can be decreased by moving from w to a nearby parameter value. SAM optimizes the sharpness in the gradient ascent direction of the current parameter point.

The Flatness Indicator Problem (FIP)

Empirically, Max-Sharpness (MaxS) behind SAM’s generalization enhancements faces the “Flatness Indicator Problem” (FIP), where SAM only considers the flatness in the direction of gradient ascent, resulting in a next minimization region that is not sufficiently flat. Because SAM is a greedy search method in nature, a better Flatness Indicator (FI) would bring a better generalization of neural networks.

Min-Sharpness and Bilateral Sharpness

To address the FIP, the paper proposes utilizing the difference between the training loss and the minimum loss over the neighborhood surrounding the current weight, denoted as Min-Sharpness (MinS). MinS measures how quickly the training loss decreases when moving from w to a nearby parameter value. The authors combine MaxS, MinS and SAM into the proposed Bilateral SAM (BSAM). Bilateral Sharpness (BilS) is defined as Lˆ(w) = L(w) Training loss + max εmax≤ρmax L(w + εmax) − min εmin≤ρmin L(w + εmin), which ensures that during training, not only does the loss decrease, but the loss on both sides of the current parameter point also becomes more balanced, leading to a flatter parameter space.

BSAM Optimization and Gradient Conflict Resolution

The gradient for updating parameters in BSAM is given by ∇wLˆ(w) = ∇wL(w) Training loss + (∇wL(w)w+εˆmax(w) − ∇wL(w) w+εˆmax ∇min L(wt + εˆmin) g min t), which promotes finding regions of local minima while encouraging locating flatter regions. To solve the Gradient Conflict Problem (GCP), where large ρ min may cause the minimum perturbation point w min to rush over the minimum point, BSAM gradually decreases ρ min as follows: ρ min t = ˇρ min + ρˆ min − ρˇ min lrt − lrmin lrmax − lrmin, (1). This gradual decrease in ρ min ensures that there is no gradient conflict between ∇wL(w) and ∇wL(w + εˆmin) in the later stages of training.

Experimental Results and Robustness

Extensive experiments demonstrate that BSAM offers superior generalization performance and robustness compared to vanilla SAM across various tasks, i.e., classification, transfer learning, human pose estimation, and network quantization. For image classification on CIFAR-100 with ResNet-18 trained for 200 epochs using the same hyperparameters as SGD and SAM baselines, BSAM achieves an accuracy of 85.51±0.14, which is better than SAM's 84.72±0.11. Furthermore, BSAM achieves a smaller maximum eigenvalue on the test set compared to SGD and SAM, indicating that it finds flatter minima. The robustness of BSAM to label noise is also demonstrated, as models optimized with BSAM maintain over 70% accuracy even when the noise rate reaches 80% for CIFAR-10 and CIFAR-100.

Conclusion

In summary, BSAM outperforms SGD and SAM on various test datasets, indicating that BSAM can improve the model’s generalization performance. Additionally, BSAM is more effective in most cases because it considers perturbations in the gradient descent direction. Overall, BSAM consistently enhances performance compared to SGD and SAM, confirming its improved generalization.

How it works

The core mechanism involves optimizing the difference between the current training loss and the minimum loss within the local parameter space, defined as Min-Sharpness (MinS). This is merged with MaxS to create Bilateral Sharpness (BilS). The update rule for weights in BSAM is derived as ∇wLˆ(w) = ∇wL(w) Training loss + (∇wL(w)w+εˆmax(w) − ∇wL(wt) w+εˆmax ∇min L(wt + εˆmin) g min t), which combines the gradient from the training loss with gradients that promote flat minima.

Parameter Studies and Convergence

The value of ρ min affects whether a parameter point remains a minimum within a local region of w after perturbation of the gradient descent direction, which impacts the gradient direction during optimization and ultimately affects model performance. The optimal results are achieved when ρ min = 0.05 for CIFAR-10. Theoretical analysis proves that BSAM converges to local minima. Theorem 1 establishes a bound for the convergence of BSAM, showing that T X−1 t=0 Egt squared ≤ L(w0) − E [L(wt+1)] Z1√T + Z1Z2√T, where Z1 and Z2 are constants that only depend on τ, ρ max, σ.

Applications

BSAM is applicable across various tasks, including classification, transfer learning, human pose estimation, and network quantization. In 2D Human Pose Estimation (2D HPE), BSAM performs better on key body joints such as Elbow, Wrist and Hip. For Quantization-Aware Training (QAT), BSAM achieved results that were higher than those of the models trained with SGD, even surpassing the accuracy of the original (unquantized) model on ResNet-18.

References

[1] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in Proceedings of the International Conference on Learning Representations (ICLR), 2021.

[8] J. Kwon, J. Kim, H. Park, and I. K. Choi, “Asam: Adaptive sharpnessaware minimization for scale-invariant learning of deep neural networks,” in Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021.

[9] J. Du, H. Yan, J. Feng, J. T. Zhou, L. Zhen, R. S. M Goh, and V Y Tan, “Efficient sharpness-aware minimization for improved training of neural networks,” in Proceedings of the International Conference on Learning Representations (ICLR), 2022.

Improvements for AI systems

Improved optimization process via Bilateral SAM (BSAM) can find flatter minima than vanilla SAM by utilizing both Max-Sharpness (MaxS) and Min-Sharpness (MinS). This allows BSAM to guide optimizers to a flatter minimum through the loss function Lˆ(w) which balances training loss with bilateral sharpness.
Enhanced generalization performance across diverse tasks by using BSAM, as experiments show it outperforms both SGD and SAM on the test datasets and achieves smaller maximum Hessian eigenvalues, indicating a finding of flatter minima.
Increased robustness to label noise in classification tasks because BSAM maintains accuracy better than SAM when label noise is high, as models trained with BSAM decrease more slowly with increasing label noise.
Better performance in transfer learning scenarios because BSAM further improves SAM and achieves optimal results across multiple datasets for both ResNet-18 and MobileNetV1, showing superior generalization compared to SGD and SAM.
Improved accuracy in quantization-aware training by using BSAM as the optimizer for QAT, where quantized models trained with BSAM achieved higher results than those trained with SGD, even surpassing the accuracy of the original unquantized model on ResNet-18.
Enhanced performance in 2D Human Pose Estimation (2D HPE) by using BSAM to optimize SimCC, as it performs better on key body joints such as Elbow, Wrist and Hip.