Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment

arXiv:2409.00880 · cs.LG · Submitted 2024-09-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment".

Jane: The gist: The proposed design methodology combines quantization, pruning,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we’ve talked about how this paper, "Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment," uses quantization, pruning, and knowledge distillation together to create leaner models that work well on embedded CPUs and GPUs two <ref:2409.00880#pg1,Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment>. What is the main message we should take away from this research?

Tom: The most important thing is that you shouldn't just look at compression techniques in isolation when you’re trying to deploy these kinds of safety monitors two <ref:2409.00880#pg1>. The combination of these three steps—quantization, pruning informed by knowledge distillation, and final pruning—is what yields a significant decrease in both memory usage and execution time while still keeping the AUROC within five percent of the baseline two <ref:2409.00880#pg2>.

Lu: They proved that you can exploit the combined effect on latent space OOD detection that wasn't explored before two <ref:2409.00880#pg1,combined effect on latent space OOD detection>. It shows how combining these methods creates a design methodology that works.

Meng: For someone shipping models to production, this methodology gives you a structured way to go about designing a detector, instead of just trying random combinations of pruning or quantization levels two <ref:2409.00880#pg1>. It’s a repeatable process.

Jane: So for listeners who might not be deep into the math—for example, someone who just wants to know if this means more reliable safety monitoring in the devices they use every day—what is that simple explanation?

Tom: It means the AI safety features in those small devices can run on their own hardware without hogging all the power or slowing things down too much two <ref:2409.00880#pg1>. It’s about making advanced detection practical for real-time use.

Lu: The work focuses on applying this methodology to existing detectors like β-VAE and optical flow, showing it works across different kinds of latent representations seven <ref:2409.00880#pg2>.

Jane: And what's the final thought on where this research might go next, or what the paper itself flags as a limitation?

Tom: They mention that the search space for this design problem is really huge because you have to choose quantization levels and pruning amounts, and they just state that there’s still a lot of work ahead in finding the optimal balance two <ref:2409.00880#pg1>. It’s not a finished solution yet.

Meng: That makes sense. We've shown one successful pathway, but the practical tuning for every new model will still require significant effort two <ref:2409.00880#pg1>.

Jane: So, to summarize, this paper provides a concrete design methodology combining three compression techniques to make VAE-based OOD detectors much smaller and faster for real-time embedded use two <ref:2409.00880#pg1>. That's the core idea of "Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment."

Conclusion: Jane: So we’ve looked at how this paper, "Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment," uses quantization, pruning, and knowledge distillation together to create leaner models that work well on embedded CPUs and GPUs two. What is the main message we should take away from this research?

Tom: The most important thing is that you shouldn't just look at compression techniques in isolation when you’re trying to deploy these kinds of safety monitors two. The combination of these three steps—quantization, pruning informed by knowledge distillation, and final pruning—is what yields a significant decrease in both memory usage and execution time while still keeping the AUROC within five percent of the baseline two.

Lu: They proved that you can exploit the combined effect on latent space OOD detection that wasn't explored before two. It shows how combining these methods creates a design methodology that works.

Tom: It means the AI safety features in those small devices can run on their own hardware without hogging all the power or slowing things down too much two. It’s about making advanced detection practical for real-time use.

Lu: The work focuses on applying this methodology to existing detectors like β-VAE and optical flow, showing it works across different kinds of latent representations seven.

Jane: And what's the final thought on where this research might go next, or what the paper itself flags as a limitation?

Aditya Bansal, Michael Yuhas, Arvind Easwaran

College of Computing and Data Science · Energy Research Institute © NTU, Interdisciplinary Graduate Program, Nanyang Technological University, Singapore

cs.LG

Submitted: 2024-09-02

Updated: 2024-09-02

Comments: Accepted to IEEE RTCSA 2024

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist: The proposed design methodology combines quantization, pruning, and knowledge distillation to create lean VAE-based OOD detectors capable of real-time inference on embedded CPUs and GPUs

Key concepts

Out-of-Distribution (OOD) Detection: This is a safety mechanism in embedded systems that identifies data samples the model was never trained on. By flagging these unseen inputs, the system can prevent potentially unsafe actions, acting as a crucial monitor against unexpected inputs.
Variational Autoencoder (VAE): A VAE is a type of generative deep learning model used here for OOD detection. It learns to compress complex data into a lower-dimensional latent space. Detection is performed by analyzing how well new samples fit within this learned latent space.
Quantization: This technique reduces the precision of the model's weights and activations, typically moving from floating-point numbers to lower bit-widths (e.g., 8-bit integers). This drastically shrinks the model size and speeds up inference time, making it suitable for resource-constrained embedded hardware.
Knowledge Distillation: This involves training a smaller 'student model' to mimic the behavior of a larger, more complex 'teacher model.' In this context, it helps create a simpler OOD detector that retains much of the performance of the original, more accurate VAE detector.

Terminology

AUROC — 2× in this paper · explained in 3 episodes
Latent Space — 2× in this paper · explained in 7 episodes
Distillation — 1× in this paper · explained in 2 episodes
Knowledge Distillation — 1× in this paper · explained in 15 episodes
Pruning — 1× in this paper · explained in 4 episodes
Quantization — 1× in this paper · explained in 15 episodes
Variational Autoencoder (VAE) — 1× in this paper · explained in 6 episodes

Summary

The gist: The proposed design methodology combines quantization, pruning, and knowledge distillation to create lean VAE-based OOD detectors capable of real-time inference on embedded CPUs and GPUs while maintaining acceptable classification accuracy.

Motivation for Compression

Out-of-distribution (OOD) detectors are crucial safety monitors in embedded cyber-physical systems by identifying samples outside a machine learning model’s training distribution to prevent potentially unsafe actions. However, OOD detectors implemented using deep neural networks often struggle to meet real-time deadlines on resource-constrained embedded systems with memory and power limitations. The research focuses on the class of variational autoencoder (VAE) based OOD detectors where detection is performed in latent space. Previous works have explored knowledge distillation, pruning, and quantization for OOD detection individually, but none have focused on the combination of all three techniques. The framework relies on three stages: 1) quantization, 2) pruning informed knowledge distillation, and 3) final pruning. This methodology generates two models: a floating point version tailored to GPU inference and a quantized version tailored to CPU inference.

Compression Techniques Applied

The compression techniques are applied in a specific sequence to achieve the best results. The search space for this design problem is prohibitively large, necessitating a structured approach

Quantization: This involves reducing the precision of weights and activations by storing tensors at a lower bit-width than floating-point precision to achieve drastic execution time improvements in model inference. Quantized networks have shown impressive performance on various benchmark datasets such as MNIST, CIFAR-10 and ImageNet.
Pruning Informed Knowledge Distillation: This technique aims to train a simpler and smaller ‘student model’ from a larger complex model or an ensemble of models with minimal loss in accuracy. The methodology proposes that all student models should be based on the teacher architecture and that execution time decreases should come from the removal of layers.
Obtaining Sparsest Model: This step involves a binary search across sparsity (from 0 − 100%) to find a model with the greatest possible sparsity that still satisfies all constraints.

Case Study Results

The methodology was demonstrated on two VAE-based OOD detectors: β-VAE and optical flow (OF). For the OF model, execution time was reduced by 20% and 28% on GPU and CPU respectively while maintaining AUROC within 5% of baseline.

Conclusion

The experiments reveal that low training loss does not necessarily translate to a high OOD accuracy. The paper concludes that having a design methodology that combines quantization, knowledge distillation, and pruning allows engineers to exploit the combined powers of these techniques instead of considering them individually.

References

[1] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R Muller, “A unifying review of deep and shallow anomaly detection,” Proceedings of the IEEE.

[2] S. Ramakrishna, Z. Rahiminasab, G. Karsai, A. Easwaran, and A. Dubey, “Efficient out-of-distribution detection using latent space of β-vae for cyber-physical systems,” ACM Transactions on Cyber-Physical Systems (TCPS), vol. 6, no. 2 <ref:2409.00880#pg2>.

[3] Q. Wu, H. Jiang, H. Yin, B. Karlsson, and C.-Y Lin, “Multi-level knowledge distillation for out-of-distribution detection in text,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL <ref:2409.00880#pg2>.

[4] S. Koda, A. Zolfit, E. Grolman, A. Shabtai, I. Morikawa, and Y. Elovici, “Pros and cons of weight pruning for out-of-distribution detection: An empirical survey,” in 2023 International Joint Conference on Neural Networks (IJCNN). IEEE <ref:2409.00880#pg2>.

[5] M. Yuhas, D. J. X. Ng, and A. Easwaran, “Design methodology for deep out-of-distribution detectors in real-time cyber-physical systems,” in 2022 IEEE 28th International Conference on Embedded and RealTime Computing Systems and Applications (RTCSA). IEEE <ref:2409.00880#pg2>.

[6] Y. Chen, B. Zheng, Z. Zhang, Q. Wang, C.

Improvements for AI systems

Knowledge Distillation for Latent Space Detection Stability + Efficiency Improvements +38% execution time on β-VAE model with only 1% AUROC drop on β-VAE with 30 latent dimensions and β=1.4, as demonstrated in the case study for the optical flow OOD detector where AUROC does not sharply fall off for the fp16 model until only one layer is left.
Quantization Aware Training (QAT) for Optimal Precision +Highest AUROC among compression techniques on β-VAE, as quantization aware training results in the highest AUROC value, among the three techniques, ensuring that low training loss does not necessarily translate to a high OOD accuracy.
Structured Layer-wise Pruning Strategy for Robust Compression +Aggressive pruning of weights and biases to achieve optimal sparsity while maintaining accuracy, as the methodology ensures that even with aggressive pruning, good OOD detection performance is maintained because mean loss increases proportionally for ID and OOD samples.
Dual Model Generation for Hardware Optimization +Creation of two tailored models—a floating point version tailored to GPU inference and a quantized version tailored to CPU inference—allowing deployment on heterogeneous embedded CPUs and GPUs with reduced execution times of 20% (GPU) and 28% (CPU) while keeping AUROC within 5% of the baseline.