Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization
stat.ML, cs.LG
Submitted: 2026-10-12
Updated: 2026-10-12
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis
- Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning
- On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
- Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
- Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks
- Analysis of Adam Algorithms for Stochastic Dynamic Systems
- Adaptive Preconditioners Trigger Loss Spikes in Adam
- FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
- Disentangling Adaptive Gradient Methods from Learning Rates
- Scalable Second Order Optimization for Deep Learning
- A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale
- 4-bit Shampoo for Memory-Efficient Network Training
- DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
- Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer
- The Convergence Behavior of Adam under Heavy-Tailed Noise
- Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
- Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $\epsilon$ via Gradient Magnitude Histogram Analysis
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier