TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

arXiv:2610.02199 · cs.LG, math.OC · Submitted 2026-10-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning".

Jane: The gist: TACO, a Ternary Absolute-max Column-wise One-sparse Optimizer,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, looking at the whole picture of this "TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning" paper, what does it ultimately suggest about how we approach memory in training large language models?

Tom: It suggests that optimization geometry itself can be a framework for designing memory efficient first-order optimizers. Instead of just trying to compress the existing state or throwing away gradients, you can derive sparsity from the fundamental mathematical structure of the update problem.

Lu: I think what this points toward is that we need to look at how the local minimization problem is formulated, like they did with min D G, D F, D (1,n→1,m) at most one), and find a geometry that naturally yields a sparse solution <ref:2610.02199#pg1>.

Meng: For deployment at scale, this means we don't have to constantly sacrifice accuracy just to fit the model size on the hardware. We get this massive reduction in memory footprint without losing competitive performance on tasks like BoolQ or RTE, which is what we need for real-world training stability and cost management.

Jane: The implication is that as models get bigger and we are limited by GPU memory, designing optimizers based on these intrinsic geometric properties might be a more powerful way to manage the trade-off between memory efficiency and accuracy than simply tuning existing hyperparameters.

Conclusion: Tom: So we're wrapping up TACO, the Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning. The authors are working on making those large language models fit on GPUs by cutting down their optimizer memory a lot.

Jane: What's the main idea here, Tom? Like, what's the actual problem they’re trying to solve?

Tom: It’s that full fine-tuning of these big models eats up way too much memory for the optimizer state. They introduce this new optimizer called TACO which cuts that memory down by a factor of one hundred seventy-four compared to AdamW8bit.

Lu: The core mechanism they are using is based on finding the exact steepest-descent direction under a specific operator norm geometry, and they get sparsity from that structure itself.

Meng: From my side, when I think about deployment, what’s important is that this memory saving happens without hurting the accuracy of the fine-tuning run. That’s usually where things get shaky.

Jane: The paper claims they maintain comparable accuracy, so what are the actual numbers on how much memory it saves?

Tom: They show a reduction in peak training memory by nearly three times, from eighty point six gigabytes down to twenty seven point five gigabytes on an OPT thirteen billion parameter model.

Lu: And they prove a non-convex convergence guarantee for this vanilla TACO, showing the iterates actually reach an epsilon stationary point in order epsilon squared iterations.

Jane: That’s huge because proving convergence is often the hardest part of new optimization methods. So, what does this mean for someone who just wants to train a model without needing a PhD in math?

Tom: It means we can push full fine-tuning of models up to thirty two billion parameters on just one eighty gigabyte H100 GPU across different families and tasks.

Meng: That ability to do that on one card is what makes it viable for real-world production runs, not just research experiments.

Jane: It seems like the focus now shifts from finding a new optimizer every time to using the geometry of optimization itself as a tool to solve these resource constraints. Where does this leave us in terms of future work?

Jichao Jiang, Cristian McGee, El Houcine Bergou, HanQin Cai, Aritra Dutta

University of Central Florida · Mohammed VI Polytechnic University

cs.LG, math.OC

Submitted: 2026-10-01

Updated: 2026-10-06

Comments: 24 pages, 7 figures, 10 tables. Code available at https://github.com/Jichao2357/TACO_optimizer

Code: https://github.com/Jichao2357/TACO

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: The gist: TACO, a Ternary Absolute-max Column-wise One-sparse Optimizer, reduces optimizer state memory by 174× relative to AdamW8bit and achieves competitive accuracy for full-parameter fine-tuning

Key concepts

TACO Optimizer: Ternary Absolute-max Column-wise One-sparse Optimizer is a first-order optimization method that computes updates based on the sign of the largest magnitude entries in columns of weight matrices. This geometric approach ensures updates are 'ternary, column-wise and one-sparse,' leading to nearly negligible persistent optimizer state memory.
Steepest Descent Direction: This is the direction in which a function decreases most rapidly at a given point. TACO computes this exact direction under a dimension-normalized 1->1 operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices, providing an exact minimization path.
Operator Norm Geometry: This refers to a specific geometric constraint used to define the optimization problem. TACO reformulates local minimization as min D <G, D>F subject to ||D||(1,n->1,m) ≤ 1. This geometry dictates that the exact minimizer is defined by D⋆ = - m/n S(G), which defines the optimizer's update rule.
Memory Efficiency Gains: The practical design retains only a small set of low-precision gradient components per column, reducing persistent state memory complexity from O(mn) to O(kn). This results in a 174x reduction in memory compared to AdamW8bit and allows full-parameter fine-tuning of 30-32B models on single high-end GPUs.

Terminology

Fine-Tuning — 1× in this paper · explained in 5 episodes

Summary

The gist: TACO, a Ternary Absolute-max Column-wise One-sparse Optimizer, reduces optimizer state memory by 174× relative to AdamW8bit and achieves competitive accuracy for full-parameter fine-tuning of large language models (LLMs) on a single 80 GB H100 GPU

How it works

TACO computes the exact steepest-descent direction under a dimension-normalized 1 → 1 operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This approach retains first-order gradients while making optimizer state memory nearly negligible. The update is derived from reformulating the local minimization problem as min D ⟨G, D⟩F, D(1,n→1,m) ≤ 1. Under this geometry, the exact minimizer is defined as D⋆ = - m/n S(G), which results in an update that is ternary, column-wise and one-sparse.

Key Theoretical Properties

The paper establishes a non-convex convergence guarantee for vanilla TACO, proving that the iterates reach an ε-stationary point in O(ε−2) iterations under standard smoothness assumptions. Furthermore, in a simplified fixed-input setting where the objective depends on W through the layer output F(W) = ϕ(W z), TACO shares the continuation geometry and limiting solution of an Adam proxy while differing from Muon. The theorem shows that W⋆TACO = W⋆Adam = W0 + (u⋆ − W0z) sign(z)⊤ z1.

Practical Design and Memory Efficiency

The practical implementation of TACO addresses the instability caused by applying the sign function directly to a single minibatch gradient by using a sparse gradient history and dynamic heavy hitters. The update rule uses Mt = Hk(µMt−1 + (1 − µ)Gt), where Hk retains the k largest-magnitude entries of every column, and only these retained values are used in the final update Dt = m/n S(Mt). This design reduces persistent state memory complexity from O(mn) to O(kn), where k is the heavy hitter threshold.

Empirical Validation at Scale

TACO has been validated across five model families (OPT, Qwen3, Pythia, Llama, and Mistral) and multiple downstream tasks (SST-2, RTE, BoolQ, COPA, CB, MultiRC). Empirical results show that TACO consistently requires the least GPU memory while maintaining competitive performance across all tasks. Specifically on OPT-13B for SST-2, TACO achieves 94.22% accuracy with 27.54 GB of peak training memory, significantly less than AdamW8bit's 80.6 GB. The optimizer step budget can be increased beyond the default 2K-step to improve performance on tasks like BoolQ and RTE.

Scaling and Transfer Capabilities

The study demonstrates that TACO enables full-parameter fine-tuning of models up to 32B parameters on a single 80 GB H100 GPU across multiple model families and tasks. The theoretical analysis confirms that TACO satisfies the standard µP spectral scaling law when the largest magnitude coordinates selected across columns remain reasonably balanced across rows, which provides a basis for learning rate transfer across model widths. Furthermore, TACO is agnostic to model architecture, achieving strong performance on Llama-3.1-8B and Qwen3-32B while maintaining a small persistent optimizer state.

Conclusion

TACO is introduced as a new first-order optimizer for memory-efficient full-parameter LLM fine-tuning that derives its sparsity from an intrinsic optimization geometry. It provides a favorable combination by inheriting the reduced hyperparameter sensitivity associated with µP scaling, following the same continuation geometry as AdamW under theoretical settings, and requiring almost no persistent optimizer state. The practical design effectively manages gradient history to maintain performance while drastically reducing memory footprint. This work suggests that optimization geometry can provide a useful framework for designing memory-efficient first-order optimizers. The paper concludes by showing that TACO maintains competitive performance with a small memory footprint, enabling full-parameter fine-tuning of models up to 32B parameters on a single 80 GB H100 GPU. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6. The complete architecture transfer results for Pythia-12B, Llama-3.1-8B, Qwen3-32B, and Mistral24B are reported and discussed in Section 6. The main performance and memory observations are discussed in Section 6.

Improvements for AI systems

Bold header: Ternary Absolute-max Column-wise One-sparse (TACO) Optimizer Implementation

This implementation retains first-order gradients while making optimizer state memory nearly negligible, reducing persistent optimizer state by 174× relative to AdamW8bit and enabling full-parameter fine-tuning of models up to 32B parameters on a single 80 GB H100 GPU.

Bold header: Enhanced Convergence Guarantees

The system can leverage the theoretical proof that vanilla TACO reaches an ε-stationary point in O(ε−2) iterations under standard smoothness assumptions, providing a rigorous guarantee for convergence during fine-tuning.

Bold header: Robust Optimization Geometry Transfer

Since TACO and Adam follow the same continuation geometry and converge to the same parameter-space solution under certain conditions, the improved system can leverage AdamW-pretrained models without suffering from optimizer mismatch, which is a known issue when switching to geometries like Muon.

Bold header: Improved Model Size Scaling

The TACO optimizer allows for full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks, enabling practitioners to scale training capacity far beyond the limits imposed by dense optimizer states.

Bold header: Task Performance Maximization via Hyperparameters

The system can utilize empirical findings that k = 16 uses only 38.3 MB of optimizer state, a 34.4× reduction from dense gradient history, suggesting that setting the heavy hitter threshold k = 16 is optimal for achieving the strongest BoolQ and RTE performance.

Bold header: Learning Rate Transfer Across Model Scales

The system can employ a shared TACO configuration across model scales because empirical results show that for every model-dataset configuration, the best learning rate is consistently found at 3 × 10−5, supporting learning rate transfer across scales rather than requiring repeated tuning.

Bold header: Architecture Agnostic Fine-tuning

The system can be used to achieve strong performance on Llama-3.1-8B and Qwen3-32B across all three tasks, demonstrating that TACO is agnostic to model architecture and maintains competitive results across different model families.

Abstract

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized 1 to1 operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by 174 times relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by 2.9 times (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.