SLIM: Semantic-based Low-bitrate Image compression for Machines

The gist: SLIM proposes Semantic-based Lowbitrate Image compression for Machines by leveraging diffusion, which focuses on Region-of-Interest (RoI) areas and uses a pretrained latent diffusion model to enhance decompressed latents, achieving high classification accuracy at low bitrates.

Proposed Method Overview

The SLIM framework is a new effective training framework of image compression for machine vision, using a pretrained latent diffusion model. The compressor model focuses only on the Region-of-Interest (RoI) areas for machine vision in the image latent to compress it compactly. Then the pretrained Unet model enhances the decompressed latent, utilizing a RoI-focused text caption which containing semantic information of the image. Therefore, SLIM is able to focus on RoI areas of the image without any guide mask at the inference stage, achieving low bitrate when compressing. And SLIM is also able to enhance a decompressed latent by denoising steps, so the final reconstructed image from the enhanced latent can be optimized for the machine vision task while still containing perceptual details for human vision.

System Architecture and Training

The overall architecture of our model is shown in Fig. 1. The encoder part consists of a VAE encoder and a compressor, and the decoder part includes a decompressor, a VAE decoder, CLIP encoder, Unet, and control module. We leverage pretrained components from Stable Diffusion 2.1 including the VAE encoder, VAE decoder, Unet, and CLIP encoder. The key trainable components of SLIM are the compressor, the decompressor, and the control module. During inference, an input image is first encoded into a latent representation by the VAE encoder. The compressor then transforms this latent into a bitstream focusing on RoI areas, and this bitstream is transmitted to the decoder along with a RoI-focused text caption extracted from the image. In the decoder, before passing through the VAE decoder, the latent is enhanced by the Unet which is conditioned on the RoI-focused text caption via the control module.

The training phase of SLIM is divided into two stages. In stage 1, we define guide loss as Lguide = MSE(MROI (z), MROI (ˆz)) where z and zˆ mean the latent encoded by the VAE encoder and the latent decompressed by the decompressor. The predefined RoI-mask used in the guide loss is obtained by Grad-CAM when the image samples pass the pretrained ResNet50. With bitrate loss R, SLIM is trained to reduce more bits on non-RoI areas where fidelity is less critical. For training the control module attached to the pretrained Unet, we define the denoising loss as Ldenoise = MSE(ϵ, ϵθ(zt, t, c)) where ϵ is the target noise and ϵθ is the predicted noise by the Unet attached with the control module with weight parameters θ. The loss function of training stage 1 is formulated as Lstage1 = λr · R + λg · Lguide + λd · Ldenoise.

Two-Stage Training Objectives

In stage 2, additional losses are included in the training stage 2. To ensure the reconstructed image samples maintain the fidelity and semantics of RoI areas in original images, we define the RoI loss as LROI = MSE(MROI (x), MROI (ˆx)) + LP IP S(MROI (x), MROI (ˆx)). The first term MSE(MROI (x), MROI (ˆz)) is the mean squared error between the original image x and reconstructed image xˆ in RoI areas for high fidelity of RoI areas in reconstructed images at the pixel level. The second term LP IP S(MROI (x), MROI (ˆx)) is the LPIPS loss between two inputs for high perceptual details of RoI areas in reconstructed images at the semantics level. To ensure the reconstructed image samples are optimized for machine vision, we define the task loss as Ltask = CE(y, yˆ) where y and yˆ are the ground truth class label and the predicted class label respectively. The loss function of training stage 2 is formulated as Lstage2 = Lstage1 + λROI · LROI + λt · Ltask.

Experimental Results and Performance

Experimental results show that SLIM achieves a higher classification accuracy in the same bits per pixel condition, compared to conventional image compression models for machines. Our SLIM model achieves the best rate-accuracy performance compared to other baselines when evaluated with both ResNet50 and ConvNeXt-T classifiers. In comparison with the models for machine vision, including MPA, TransTIC, AdaptICMH, our approach requires consistently fewer bits per pixel (bpp) to achieve the same classification accuracy. Furthermore, in comparison with the model for human vision, OSCAR, our approach still achieves better rate-accuracy performance. Particularly at low bitrates, SLIM maintains reasonable accuracy while OSCAR shows a significant drop in accuracy. The corresponding BD-Rate comparison results are reported in Tab. 1.

Ablation Studies and Efficiency

The diffusion module has a significant impact on the rate-accuracy performance of SLIM, achieving higher accuracy at the same bitrate level, especially in low bitrates. Without the diffusion module, no denoising steps of Unet for enhancing the image latent is involved before passing it to the VAE decoder and nonenhanced latent which decompressed by decompressor are directly used to be decoded by the VAE decoder. The diffusion module is a crucial component that enhances the imperfect latent into the latent with rich semantics and details through denoising steps of Unet. RoI-focused training leads to a higher BD-Rate performance as shown in Tab. 2, resulting in a BD-rate reduction of 18.87%. Processing a 512 × 512 sized RGB image takes a total of 1657.83 GFLOPs, with the majority of the computational cost arising from the diffusion module in the decoding phase. Our method yields improved rate-accuracy performance at the cost of increased computational complexity compared to non-diffusion based approaches. Our method achieves a decoding time comparable to OSCAR while requiring only about twothirds of its GFLOPs.

Conclusion and Future Work

In this paper, we proposed a novel semantic-based framework of learned image compression for machine vision, named SLIM. Our SLIM model could efficiently reduce the bitrate by the compressor focusing on the RoI areas of the image latent, and effectively improve the quality of a reconstructed image by the diffusion module enhancing the decompressed latent through denoising steps. This work provides a novel scheme for learned image compression in machine vision applications, utilizing RoI-focused training and pretrained diffusion. In future work, we aim to accelerate our model by utilizing an one-step diffusion.

Supplementary Information

To obtain RoI-focused text captions, we feed the RoIMasked image into BLIP2 which generates a description of the preserved areas. Samples of the resulting RoIfocused text captions after compression with the zlib lossless compression algorithm require only 367 bits on average. On this scale, the captions require only 0.0014 bpp on average, which is extremely small compared to the bitrate required for the compressed image data. We examine the effect of the training dataset size by additionally training our SLIM model on an extended dataset constructed by combining OpenImages v6 (train), CLIC2020 professional (train), and COCO2017 (train) which comprises of approximately 170,000 images. The experimental result indicates that our SLIM model has the potential to achieve better classification accuracy at low bitrates when trained on extended datasets.

Computational Cost Details

Table 3 shows the computational cost of each model for a 512 × 512 sized RGB image. The encoding phase requires 575.61 GFLOPs, while the decoding takes 1082.22 GFLOPs, indicating that the majority of the computational cost arises from the diffusion module in the decoding phase. SLIM (Ours) has a total processing time of 50.60 ms for encoding and 212.95 ms for decoding. The overall complexity could be further reduced by employing a onestep diffusion model.

Additional Samples

Figure 11a shows reconstructed images comparing the SLIM models with and without the diffusion module. Figure 11b shows the bit allocation maps obtained from the SLIM models with and without RoI-focused training.

1. Semantic-based Lowbitrate Image compression for Machines SLIM can achieve higher classification accuracy in machine vision tasks at a lower bitrate compared to conventional image compression models for machines, as shown by achieving a higher classification accuracy in the same bits per pixel condition (Page 1).

2. The system can effectively reduce the bitrate by focusing on the Region-of-Interest (RoI) areas relevant to machine vision tasks, allowing low bitrate compression without significant loss of important information (Page 2).

3. The reconstructed images from SLIM can be optimized for machine vision tasks while still containing perceptual details for human vision because the system uses denoising steps of the diffusion model enhance the decompressed latent representations into refined latent features (Page 1).

4. SLIM can maintain reasonable accuracy at low bitrates where other models like OSCAR show a significant drop, suggesting it is specially designed for machine vision and its effectiveness is increasingly noticeable when a compression ratio increases (Page 4).

5. The system demonstrates superior rate-accuracy performance compared to baselines, achieving an average BD-Rate reduction of-94.56 % against the MPA model on the ImageNet-val dataset (Page 4).

6. The model can preserve crucial texture and details in RoI areas while strategically removing details only in non-RoI areas, leading to reconstructed images with more natural textures than others, particularly in the RoI areas for machine vision (Page 6).