FeaTok: Feature-Loss Tokenization for Image Generation

The gist The authors introduce Feature-Loss Tokenization (FeaTok), a discrete image tokenizer that trains tokens by reconstructing pretrained visual features instead of pixels, which advances the quality–compute Pareto frontier for image generation

Motivation and Background

Discrete image generators learn the distribution of tokens produced by an image tokenizer, and these token representations should be easy for generators to model, not merely enable faithful image reconstruction. Previous discrete tokenizers focus on reconstruction performance, which targets the first property, expressivity. However, generation also depends on the second property, which they call generative learnability: how easily a generator fits the token distribution at a given budget. Generative learnability is related to the structure of a representation (Skorokhodov et al., 2025), which is in turn induced by the training target. Pixel reconstruction prioritizes high-variance image details over features useful for perception (Balestriero and Lecun, 2024), and using self-supervised learning (SSL) objectives to predict features instead of pixels yields more semantically meaningful representations (Assran et al., 2023).

FeaTok Architecture

FeaTok separates discrete representation learning from image reconstruction by using a feature decoder that reconstructs pretrained visual features rather than pixels The pipeline involves an encoder E, a quantizer Q, and a feature decoder Dfeat that maps the discrete tokens to a prediction yˆ1:N of the feature target y1:N. The training objective Lfeat combines l1, l2, and cosine distances to match this prediction to the target using a loss Lfeat. FeaTok also incorporates global augmentation by discretizing the encoder’s global representation (e.g., CLS token) and generating it before the patch tokens, giving them image-level context. This results in a factorization of the conditional distribution pϕ(zglobal, z1:N c) = pϕ(zglobal c) Y N i=1 pϕ(zi zglobal, z<i, c).

Training and Loss Functions

The complete objective LFeaTok combines feature and pixel losses with a regularization term Lreg between the last-layer features of E and Eˆ. The loss is defined as LFeaTok = Ex [Lfeat(Dfeat(z), y) + λregLreg(h, hˆ) z] which trains encoder, quantizers, feature decoder + λpixLpix(Dpix(sg(z1:N)), x) which trains only the pixel decoder. The pixel losses Lpix combine pixel, perceptual, Gram, and adversarial losses with an alignment loss that regresses the frozen last-layer features from the third pixel-decoder block through an MLP. Crucially, stop gradients at the decoder input preserves the ability to decode tokens back into images while preventing pixel reconstruction losses from shaping the token distribution.

Performance and Results

FeaTok demonstrates superior generative performance across various benchmarks. Ablations across SigLIP, SigLIP2, DINOv2, and DINOv3 show better generative performance with FeaTok. In class-conditional generation on ImageNet-1k 256×256, FeaTok advances the quality–compute Pareto frontier for image generation by reducing gFDr6 at 256×256 by up to 29% over the strongest discrete baselines and outperforming the strongest unguided continuous baseline (RAEv2) with 38× less sampling compute. FeaTok also achieves higher human-preference reward-model scores than prior art without reward-based training and surpasses continuous and discrete text-conditional baselines on three benchmarks with only 500 fine-tuning updates after pre-training. In comparison to continuous models at a matched sampling cost, FeaTok is competitive, achieving gFDr6 3.13 for FeaTok-L-DINOv3 without guidance, outperforming RAEv2 (3.95) while using 1.27 rather than 47.72 TFLOPs per image.

Conclusion

FeaTok is a discrete image tokenizer whose tokens are shaped only by feature reconstruction, while a pixel decoder trained on detached tokens renders them into images. Across four pretrained encoders, FeaTok generates better than tokenizers trained with pixel or combined losses. FeaTok achieves the best gFDr6 among the evaluated discrete generators on ImageNet at 256×256 and 512×512, outperforms continuous generators at a matched sampling cost, achieves higher human-preference reward-model scores, and transfers to text-to-image generation after pretraining and only 500 fine-tuning updates. The low sampling cost of FeaTok makes it attractive for scaling multimodal training.

How it works

1. Conventional discrete tokenizers consist of an encoder E, a quantizer Q, and a pixel decoder Dpix, typically trained as an autoencoder with a pixel-level objective Lpix that combines pixel reconstruction losses (l1, l2), perceptual losses (LPIPS (Zhang et al., 2018), GRAM (Lu et al., 2026a)), and adversarial losses.

2. FeaTok learns discrete tokens by reconstructing features rather than pixels, adding a feature decoder Dfeat that maps the discrete tokens to a prediction yˆ1:N of a feature target y1:N.

3. The feature target is an average of features from multiple layers of the frozen vision encoder Eˆ, y1:N (x) = K−1 P k∈K LNk(Eˆ(k) (x)1:N), where K contains four evenly spaced encoder layers and LayerNorm (Ba et al., 2016) normalizes each layer’s features before averaging.

4. FeaTok leverages global tokens by discretizing the encoder’s global representation along with patches, allowing for a hierarchical generative model that first generates the global token and then the patch tokens conditioned on it.

Controlled Ablations

The authors verify FeaTok against pixel-loss tokenization using four pretrained vision encoders: SigLIP (Zhai et al., 2023), SigLIP2 (Tschannen et al., 2025), DINOv2 (Oquab et al., 2024), and DINOv3 (Siméoni et al., 2026). gFDr6 averages Fréchet distances over six representation spaces, each normalized by the distance between the training and validation sets. Human-preference rewards are measured using PickScore (Kirstain et al., 2023), HPSv2 (Wu et al., 2023), and HPSv3 (Ma et al., 2025). The results are reported on ImageNet-1k at both 256×256 and 512×512 resolutions.

Token Structure Measurement

The spatial dependence diagnostic measures how informative one patch token is about an adjacent patch token in the same image by estimating MI between pairs of bits from their joint occurrence frequencies across training images The feature target raises this dependence for SigLIP, DINOv2, and DINOv3, but it lowers the dependence for SigLIP2.

Image Generation Results

FeaTok outperforms all discrete baselines with and without guidance in gFDr6 across all four encoders on ImageNet-1k 256×256 and 512×512.

1. Bold header: Feature-loss tokenization for discrete representation learning; FeaTok train[s] discrete tokens by reconstructing pretrained features rather than pixels. This enables generators to model representations that are easy for generators to model, not merely enable faithful image reconstruction, which improves generative performance over pixel-based or mixed losses.

2. Bold header: Advancement of the quality–compute Pareto frontier; FeaTok advances the quality–compute Pareto frontier for image generation, reducing gFDr6 at 256x256 by up to 29% over strong discrete baselines and outperforming the strongest unguided continuous baseline (RAEv2) with 38× less sampling compute.

3. Bold header: Superior generative performance across multiple encoders; FeaTok has better spatial structure and generates better than tokenizers trained with pixel or combined losses for all four encoders we study, including SigLIP, SigLIP2, DINOv2, and DINOv3.

4. Bold header: Enhanced human preference alignment without reward training; FeaTok achieves higher human-preference reward-model scores than prior art without reward-based training. This allows for the development of models that are closer to those of the training set on benchmarks like HPSv3, indicating better alignment with human perception.

5. Bold header: Robust transferability to text-to-image generation; FeaTok reusing the ImageNet-trained tokenizer, FeaTok surpasses the baselines on three text-to-image benchmarks after 75K pretraining updates and only 500 fine-tuning updates, demonstrating that feature-loss tokens can be used for unified multimodal models.

6. Bold header: Efficient inference at low compute cost; FeaTok is competitive with continuous representations at lower sampling cost. For example, FeaTok-L-DINOv3 achieves gFDr6 3.13 at 256x256 with a TFLOPs of 1.27, significantly lower than continuous baselines like RAEv2 (47.72 TFLOPs).

Discrete image generators learn the distribution of tokens produced by an image tokenizer. These token representations should be easy for generators to model, not merely enable faithful image reconstruction. Yet these tokenizers are commonly trained to reconstruct pixels, which prioritizes low-level fidelity and may disrupt representation structure useful for generation. Motivated by feature-predictive self-supervised learning, we introduce feature-loss tokenization (FeaTok), which trains discrete tokens by reconstructing pretrained features rather than pixels. Ablations across SigLIP, SigLIP2, DINOv2, and DINOv3 show better generative performance with FeaTok. In class-conditional generation, FeaTok advances the quality-compute Pareto frontier, reducing gFDr6 at 256x256 by up to 29 percent over the strongest discrete baselines and outperforming the strongest unguided continuous baseline (RAEv2) with 38 times less sampling compute. FeaTok also achieves higher human-preference reward-model scores than prior art without reward-based training and surpasses continuous and discrete text-conditional baselines on three benchmarks with only 500 fine-tuning updates after pre-training.