Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging

arXiv:2602.04805 · cs.GR, cs.AI · Submitted 2026-02-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging".

Jane: The gist: SkinTokens introduce a learned, compact, and discrete representation for skinning weights that reframes skinning from a continuous regression problem to a discrete token prediction task.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging. Jane, what’s the main takeaway from that title to someone just listening?

Jane: Well, it’s saying they are introducing something called SkinTokens. It sounds like they found a way to represent skinning weights in a much smaller and more organized way than the huge matrices we usually deal with.

Tom: Exactly. It's about taking this skinning problem, which is usually treated like a continuous regression task, and turning it into something discrete, something token-based that an autoregressive model can predict.

Lu: From my perspective, it seems like they are tackling the fundamental representation issue that makes automated rigging so hard right now. They’re trying to move away from just fitting fixed templates or using methods like MST algorithms which are sensitive to noise and end up with messy results, as mentioned in DRiVE Sun et al.

Meng: But what does this actually mean for us at the production level? If we can stop dealing with these massive regression outputs and instead predict a compact token sequence, that sounds like it could make the pipeline much more scalable.

Lalam: I’m seeing that they use a Finite Scalar Quantized Variational Autoencoder, or FSQ-CVAE, to compress those sparse skinning weights into discrete tokens LD = FSQ(LW) >.

Jane: So they are taking the continuous weights and squashing them into these discrete tokens first, which makes it easier for the rest of the system to handle.

Tom: Right. The authors say this representation allows them to robustly generate high-fidelity skeletons and precise skinning maps for complex geometries, ranging from stylized characters to quadrupeds Zhang et al.

Lu: I think the key is that this discrete step simplifies the problem into a token prediction task, which is what the TokenRig framework then uses.

Meng: If it’s discrete tokens, does that mean we get better control over the deformation shapes? That’s something I worry about when we have to debug these things later.

Lalam: The paper suggests this transformation into a token prediction task is what allows them to build TokenRig, which is their unified framework >.

The paper's summary: Tom: So the core idea of Skin Tokens is this learned, compact, and discrete representation for skinning weights that reframes skinning from a continuous regression problem to a discrete token prediction task. Jane, how do you explain that shift in simple terms?

Jane: Imagine instead of trying to perfectly draw every single tiny weight for every joint on the surface—which is incredibly messy—you learn a set of representative "tokens" that describe the deformation.

Tom: Right, so it’s replacing a massive matrix prediction with predicting a sequence of these compact tokens. The authors use this discretization via FSQ to get those tokens LD = FSQ(LW) >.

Lu: What I find really interesting is how they then concatenate those discrete tokens with the shape features from EM and pass them into a decoder, which is the next step in their unified model >.

Meng: That sounds like it’s trying to bridge the gap between knowing where the bones are and knowing how they deform on the surface, but it’s still a lot of moving parts. What about the training?

Lalam: The TokenRig framework itself is an end-to-end autoregressive system that learns to generate a single, coherent sequence comprising both the complete skeleton followed by these learned SkinTokens >.

Tom: So they are generating the skeleton and the skinning representation simultaneously in one go, which lets them learn those complex cross-modal dependencies between skeletal placement and surface skins.

Jane: It means the model isn't just learning a static mapping; it’s learning how different bone positions influence specific token predictions on the mesh, which is what professional rigging artists actually do.

Lu: The Transformer’s self-attention mechanism is what lets it access all joint positions and bone types when predicting the skinning for any given bone, modeling those complex long-range dependencies >.

Meng: That sounds powerful theoretically, but on practical terms, how do we know this token sequence isn't just a random mess when we deploy it?

The paper's improvements: Tom: Moving on to what they actually improved, the paper introduces a few things. The first big thing is that they use Reinforcement Learning for post-training refinement using Group Relative Policy Optimization, or GRPO >.

Jane: That sounds like adding an extra training phase where the model learns from explicit geometric and semantic rewards instead of just relying on standard supervised loss functions.

Tom: Right, and they define four specific rewards: Volumetric Joint Coverage Rvj, Bone-Mesh Containment Rvk, Skinning Coverage and Sparsity Rsc, and Deformation Smoothness Rmo >.

Lu: Those rewards are interesting because they force the model to consider things like making sure the skinning covers the whole joint volume or that it stays within bounds of the mesh.

Meng: I’m skeptical about those explicit rewards if we don't have a way to reliably measure them during training, but if it helps generalization, then maybe it’s worth trying.

Lalam: The authors state that this RL stage allows TokenRig to generalize far beyond its training data, successfully rigging complex, “in-the-wild” assets where purely supervised methods often fail >.

Jane: That means if you train it on a few characters, you can actually use it on something completely new and unexpected without retraining everything from scratch.

Tom: And the experimental validation shows that this representation leads to a ninety-eight percent to one hundred thirty-three percent improvement in skinning accuracy over state-of-the-art methods using Intersection over Union and L1 Error metrics >.

Lu: The fact that the reconstruction fidelity is concentrated in just the top few tokens, which is what they show with IoU, suggests that a lot of the critical deformation information is really packed into those most important tokens >.

Meng: If we can get ninety-eight percent accuracy on complex assets, that’s good for production, but I need to know how much computational overhead this extra RL refinement adds to the inference speed.

Lalam: The paper also notes that if the generated token sequence is invalid, it receives a reward of zero during this RL phase >.

Conclusion: Tom: So to wrap up, Skin Tokens introduces a learned compact representation via FSQ-CVAE to turn skinning into a discrete token prediction task, and TokenRig generates the skeleton and tokens together in an autoregressive sequence >.

Jane: And they’ve added Reinforcement Learning with GRPO using specific geometric rewards like volumetric joint coverage to help it generalize to complex, real-world assets >.

Tom: The results show a ninety-eight percent to one hundred thirty-three percent improvement in skinning accuracy over existing methods, and the key is that the relevant information is concentrated in just the top few tokens >.

Lu: This approach moves us away from methods that rely on fixed templates or MST algorithms which are sensitive to noisy predictions, moving toward a unified framework for rigging >.

Meng: I still have my questions about how we deploy this at scale and the cost of running that whole sequence, but the ability to handle out-of-distribution assets is definitely something worth investigating >.

Lalam: The core insight of Skin Tokens is transforming the ill-posed regression of sparse skinning matrices into a robust token prediction task >.

Jane: It’s a major step toward making automated rigging closer to what professional artists do, and it’s certainly worth keeping an eye on for future work.

Tom: Yep. So that's our take on Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging >.

Tsinghua University

cs.GR, cs.AI

Submitted: 2026-02-04

Updated: 2026-10-19

Comments: 14 pages, 10 figures

Journal ref: ACM Transactions on Graphics 2026

DOI: 10.1145/3859874

Project page: https://zjp-shadow.github.io/works/SkinTokens

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist: SkinTokens introduce a learned, compact, and discrete representation for skinning weights that reframes skinning from a continuous regression problem to a discrete token prediction task.

Key concepts

SkinTokens
A compact, discrete representation learned via a Finite Scalar Quantized Variational Autoencoder (FSQ-CVAE). Instead of regressing every skinning weight individually, this method compresses the high-dimensional sparse skinning matrix into a small sequence of discrete tokens.
TokenRig Framework
An autoregressive model that generates a single interleaved sequence containing both skeletal parameters and their corresponding SkinTokens. This allows the model to learn how the placement of bones directly influences the resulting surface deformation, capturing complex cross-modal relationships.
Reinforcement Learning Refinement (GRPO)
A post-training stage using Group Relative Policy Optimization (GRPO) guided by geometric and semantic rewards like Volumetric Joint Coverage. This technique helps the model generalize to new, unseen assets by optimizing its output against explicit rules for a valid and plausible rig.
Finite Scalar Quantization (FSQ)
A technique applied to latent features within the VAE to discretize continuous values into finite scalar tokens. This step is crucial for creating the compact, discrete representation that makes skinning prediction tractable.

Terminology

Summary

The gist: SkinTokens introduce a learned, compact, and discrete representation for skinning weights that reframes skinning from a continuous regression problem to a discrete token prediction task.

Introduction and Motivation

The rapid proliferation of generative 3D models has created a critical bottleneck in animation pipelines: rigging The rapid proliferation of generative 3D models has created a critical bottleneck in animation pipelines: rigging > Existing automated methods are fundamentally limited by their approach to skinning, treating it as an illposed, high-dimensional regression task that is inefficient to optimize and is typically decoupled from skeleton generation

SkinTokens Representation

The core of the framework is SkinTokens, a learned, compact, and discrete representation for skinning weights This representation transforms the intractable problem of regressing a high-dimensional, sparse matrix into a tractable token prediction task The authors employ a Finite Scalar Quantized Variational Autoencoder (FSQ-CVAE) to compress the sparse skinning weights for each individual bone into a compact, discrete representation This discretization step is performed by applying Finite Scalar Quantization (FSQ) to the latent features The resulting discrete tokens LD = FSQ(LW) are then concatenated with the shape features from EM and passed to a decoder

TokenRig Framework

TokenRig is a unified, end-to-end autoregressive framework that learns to generate a single, coherent sequence comprising the complete skeleton followed by the learned SkinTokens The model generates a single, interleaved sequence of skeletal parameters and their corresponding SkinTokens This holistic formulation allows the model to learn the complex, cross-modal dependencies between skeletal placement and surface skins The Transformer’s self-attention mechanism can access all joint positions and bone types when predicting the skinning for any given bone, enabling it to model complex, long-range dependencies

Reinforcement Learning Refinement

To improve generalization to out-of-distribution (OOD) assets, the authors introduce a post-training refinement stage using Reinforcement Learning (RL) They leverage Group Relative Policy Optimization (GRPO) to optimize the model directly against a set of explicit, non-differentiable geometric and semantic rewards The authors introduce four specific rewards: Volumetric Joint Coverage (Rvj), Bone-Mesh Containment (Rvk), Skinning Coverage and Sparsity (Rsc), and Deformation Smoothness (Rmo) This allows TokenRig to generalize far beyond its training data, successfully rigging complex, “in-the-wild” assets where purely supervised methods often fail

Experimental Validation

Quantitative results show that the SkinTokens representation leads to a 98%–133% improvement in skinning accuracy over state-of-the-art methods The reconstruction fidelity is evaluated using Intersection over Union (IoU) and L1 Error metrics, demonstrating that the relevant information is effectively concentrated in the top few tokens Furthermore, the RL stage ensures that if the generated token sequence is invalid, it receives a reward of 0 The framework consistently outperforms leading learning-based rigging approaches across skeletal generation and skinning prediction metrics

Conclusion

The work introduces TokenRig, an automated framework for skeletal rigging and skinning weight prediction that approaches the fidelity of professional artist workflows The core insight is that the long-standing bottleneck in automatic skinning is a representation problem By proposing SkinTokens, i.e., a compact, discrete representation learned via an FSQ-CVAE, they successfully transformed the ill-posed regression of sparse skinning matrices into a robust token prediction task This design enabled them to train a unified autoregressive model that jointly synthesizes skeletal structures and skinning weights, capturing the intrinsic dependencies between articulation and deformation that prior decoupled methods ignore The integration of geometric and semantic reward functions proves particularly effective for generalization, allowing the model to generate coherent rigs for complex, out-of-distribution assets that defy standard supervised learning

Limitations and Future Work

While the FSQ-CVAE offers high compression efficiency, a comparative analysis suggests a residual performance gap remains compared to continuous-latent VAEs in extremely challenging skinning scenarios Future work could explore physics-based rewards to further ensure the dynamic plausibility of the generated deformations during animation Additionally, extending the autoregressive model to accept user-specified topological templates or interactive guidance would be a valuable direction The paper concludes by noting that simulating topological imperfections during training is vital for handling the diverse and often noisy geometry found in production environments

References

Nadine Abu Rumman and Marco Fratarcangeli. 2015. Position-based skinning for soft articulated characters. In Computer Graphics Forum, Vol. 34, 240–250 Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. 2025. FlexTok: Resampling Images into 1D Token Sequences of Flexible Length Ilya Baran and Jovan Popović. 2007. Automatic rigging and animation of 3d characters Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation Kiran Bhat, Nishchaie Khanna, Karun Channa, Tinghui Zhou, Yiheng Zhu, Xiaoxia Sun, Charles Shang, Anirudh Sudarshan, Maurice Chu, Daiqing Li, et al. 2025. Cube: A roblox view of 3D intelligence Sue Blackman. 2014. Lazy Bones Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model Oren Rippel, Michael Gelbart, and Ryan Adams. 2014. Learning ordered representations with nested dropout John Schulman. 2020. Approximating kl divergence Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models Md Fahim Sikder, Resmi Ramachandranpillai, and Fredrik Heintz. 2025. Transfusion: generating long, high fidelity time series using diffusion models with transformers Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models Chaoyue Song, Xiu Li, Fan Yang, Zhongcong Xu, Jiacheng Wei, Fayao Liu, Jiashi Feng, Guosheng Lin, and Jianfeng Zhang. 2025a. Puppeteer: Rig and Animate Your 3D Models <ref:2602.

Improvements for AI systems

  1. textbf Learned Discrete Representation for Skinning Weights (SkinTokens): The system can transform skinning from a high-dimensional regression task into a compact sequence prediction problem. This allows for scalability and robustness by avoiding direct regression of a massive, high-dimensional N × J matrix.

  2. textbf Unified Autoregressive Framework (TokenRig): The framework can generate a single, coherent sequence that interleaves skeletal parameters with their corresponding SkinTokens, enabling the model to learn the complex, cross-modal dependencies between skeletal placement and surface skins.

  3. textbf Reinforcement Learning for Rig Refinement (GRPO): The system can generalize to out-of-distribution assets by using Group Relative Policy Optimization (GRPO) guided by rewards like Volumetric Joint Coverage and Bone-Mesh Containment, ensuring the generated rigs are topologically valid and functionally robust.

  4. textbf Enhanced Skinning Fidelity via Sparse Loss Functions: The system can achieve better reconstruction of active deformation regions by using Dice Loss, as it provides a stronger gradient signal for non-zero weights compared to BCE.

  5. textbf Robust Data Augmentation Pipeline: The model can handle diverse and noisy geometry by simulating imperfections through augmentation, as demonstrated by the finding that removing any single module results in a consistent degradation of skeletal prediction accuracy across all datasets.

Abstract

The rapid proliferation of generative 3D models has created a critical bottleneck in animation pipelines: rigging. Existing automated methods are fundamentally limited by their approach to skinning, treating it as an ill-posed, high-dimensional regression task that is inefficient to optimize and is typically decoupled from skeleton generation. We posit this is a representation problem and introduce SkinTokens: a learned, compact, and discrete representation for skinning weights. By leveraging an FSQ-CVAE to capture the intrinsic sparsity of skinning, we reframe the task from continuous regression to a more tractable token sequence prediction problem. This representation enables TokenRig, a unified autoregressive framework that models the entire rig as a single sequence of skeletal parameters and SkinTokens, learning the complicated dependencies between skeletons and skin deformations. The unified model is then amenable to a reinforcement learning stage, where tailored geometric and semantic rewards improve generalization to complex, out-of-distribution assets. Quantitatively, the SkinTokens representation leads to a 98%-133% percents improvement in skinning accuracy over state-of-the-art methods, while the full TokenRig framework, refined with RL, enhances bone prediction by 17%-22%. Our work presents a unified, generative approach to rigging that yields higher fidelity and robustness, offering a scalable solution to a long-standing challenge in 3D content creation.

Sources

Related papers