A Composite Activation Function for Learning Stable Binary Representations

summary

Video file (mp4)

The gist

The gist: This paper proposes Heavy-Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function, which enables stable training with standard gradient-based optimization for

In short

The paper introduces Heavy-Tailed Activation Function (HTAF), a smooth approximation of the Heaviside function that allows neural networks to be trained stably using standard gradient descent, overcoming issues with non-differentiable binary activations. HTAF preserves a large gradient near zero while allowing controlled decay in the tail regions. This enables stable training for Spiking Neural Networks, Binary Neural Networks, and Deep Heaviside networks.

Key concepts

Heavy Tailed Activation Function (HTAF)
HTAF is a composite function designed to smoothly approximate the sharp Heaviside step function. It is constructed using a sigmoid and a hyperbolic tangent combination. Its key feature is that it maintains a large gradient near zero for stable training, while its tail region gradient decay can be controlled by parameters.
Gradient Mass Preservation
This property refers to HTAF's ability to keep the gradient magnitude high around zero inputs. This is crucial because standard optimization methods struggle with non-differentiable functions like the Heaviside function, which have zero derivative everywhere except at the step. HTAF ensures a usable gradient exists near this critical point.
Implicit Concept Bottleneck Models (ICBMs)
ICBMs are an interpretable model extension that uses HTAF to create discrete feature representations. Instead of relying on pre-defined concepts, ICBMs learn implicit concepts directly from data using HTAF. These learned features can then be interpreted by external models like Large Language Models, yielding class-specific binary representations.
Binary Neural Networks (BNNs)
BNNs are neural networks trained to produce binary outputs instead of continuous values. Training these networks is challenging because the Heaviside activation function is not differentiable. HTAF provides a stable solution by offering a smooth, differentiable approximation that allows for effective gradient-based optimization during the training process.

Terminology used across episodes

This episode discusses

The paper

A Composite Activation Function for Learning Stable Binary Representations · Read on arXiv

Seokhun Park, Choeun Kim, Kwanho Lee, Sehyun Park, Insung Kong, Yongdai Kim

Seoul National University · University of Twente

Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "A Composite Activation Function for Learning Stable Binary Representations".

Jane: Here's the gist of the paper. The gist: This paper proposes Heavy-Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function, which enables stable training with standard gradient-based optimization for neural networks with binary representations, and introduces Implicit Concept Bottleneck Models (ICBMs) that leverage HTAF to induce discrete feature representations.

Tom: Heavy-Tailed Activation Function (HTAF) The authors propose Heavy-Tailed Activation Function (HTAF), which is defined as HTAF(xα0, α1):= σ(α1ψ(α0x)) (Page 3).

Jane: This function is constructed as a sigmoid–hyperbolic tangent composite function to approximate the Heaviside function while ensuring stable training with gradient-based optimization (Page 2).

Tom: The key idea behind HTAF is to design an activation function that "preserves a large gradient near zero (to approximate the Heaviside function closely), while allowing the gradient to decay slowly for large-magnitude inputs (to avoid the gradient vanishing problem)" (Page 2).

Paper summary: Tom: So, the paper "A Composite Activation Function for Learning Stable Binary Representations" introduces HTAF as a smooth approximation to the Heaviside function, constructed from a sigmoid hyperbolic tangent composite function <ref:2605.11558#pg3>.

Jane: And what it does is enable stable training with gradient-based optimization for neural networks that use binary representations, which was previously very challenging because the Heaviside function isn't differentiable <ref:2605.11558#pg2>.

Lu: The key idea they are pushing is designing an activation function that keeps a large gradient near zero inputs while allowing the gradient to decay slower in the tail regions, which lets them control the rate of gradient decay by adjusting alpha zero and alpha one <ref:2605.11558#pg2>.

Meng: This means for deployment, we have a more stable training method for binary models that doesn't require surrogate gradients like STE <ref:2605.11558#pg3>.

Tom: And the introduction of Implicit Concept Bottleneck Models, or ICBMs, shows this approach can induce binary latent representations corresponding to implicit concepts while retaining high prediction accuracy <ref:2605.11558#pg3>.

Jane: It’s about taking those efficient binary features and making them interpretable by showing how they relate to learned implicit concepts, which is a major step for AI understanding.

Lu: The paper concludes that HTAF is a novel activation function that stably approximates the Heaviside function under gradient-based optimization by preserving gradient magnitude <ref:2605.11558#pg5>.

Meng: It provides an effective way to train DNNs with the Heaviside activation function using a standard gradient descent algorithm without surrogate gradients, while significantly improving training stability and prediction performance <ref:2605.11558#pg3>.

Tom: That’s the core idea: HTAF is a novel activation function that stably approximates the Heaviside function under gradient-based optimization by preserving gradient magnitude <ref:2605.11558#pg5>.

Jane: And the introduction of ICBMs demonstrates that this approach can induce binary latent representations corresponding to implicit concepts while retaining high prediction accuracy <ref:2605.11558#pg3>.

Conclusion: Tom: So, we’re wrapping up this deep dive on Heavy Tailed Activation Function, or HTAF. This paper is about how they built a smooth way to approximate the Heaviside function for training neural networks with binary representations <ref:2605.11558#pg3>.

Jane: Right, so it’s about taking something that isn't differentiable and making it usable with standard gradient methods. The authors, they built this HTAF as a sigmoid hyperbolic tangent composite function <ref:2605.11558#pg2>.

Lu: The core idea they push is controlling the gradient behavior. They designed it so that near zero inputs, the gradient stays large enough to approximate that sharp Heaviside jump, but in the tail regions, it decays slower <ref:2605.11558#pg2>.

Meng: From a deployment standpoint, what matters is stability at scale. If you’re running binary models on production hardware, you need training that doesn't blow up because of tiny numerical issues <ref:2605.11558#pg3>.

Tom: Exactly. And they show that by tuning those two parameters, alpha zero and alpha one, you can manage that gradient decay in the tail regions without messing up the gradient near zero inputs <ref:2605.11558#pg2>.

Jane: That means for networks using binary representations—like spiking or deep Heaviside neural networks—you can train them stably using standard gradient descent, not needing those extra tricks like surrogate gradients <ref:2605.11558#pg3>.

Lu: And they show this works across several network types, including spiking neural networks and binary neural networks <ref:2605.11558#pg3>. It's a solid theoretical foundation for working with those discrete representations.

Meng: I’m curious about the application side. They introduce Implicit Concept Bottleneck Models, or ICBMs, which use this HTAF to induce discrete feature representations <ref:2605.11558#pg2>.

Tom: That’s where it gets interesting for interpretation. ICBMs use HTAF in the final layer during training to find these binary concepts, and then at inference time, they switch back to the Heaviside function for truly discrete outputs <ref:2605.11558#pg2>.

Jane: So it’s not just making them train better; it’s about making those features inherently interpretable by linking them to learned concepts <ref:2605.11558#pg7>.

Lu: They claim this allows the AI to learn these implicit concepts directly from the data, instead of asking a human to label things first <ref:2605.11558#pg7>.

Meng: And in their experiments, they show that ICBMs achieve prediction performance that’s comparable to or even better than standard image models on datasets like CIFAR-ten and CIFAR-one hundred <ref:2605.11558#pg32>. That's a big win for practical deployment.

Tom: So the authors are basically saying they found a way to get the benefits of binary representations—efficiency and interpretability—while keeping the performance high with this HTAF activation <ref:2605.11558#pg4>.

Jane: It’s a lot of moving parts, though. They do mention that tuning alpha zero and alpha one helps with vanishing gradients, but they also flag that you need to be careful; if you tune them too much, the numerical stability can actually get worse <ref:2605.11558#pg5>.

Lu: That’s a fair caveat. They show the limit of the derivative is mostly governed by alpha zero, but at the origin, it depends on both alpha zero and alpha one <ref:2605.11558#pg32>. It’s a bit nuanced there.

Meng: For me, what I'm hearing is that if you want to build efficient binary models that actually work reliably in the real world, this HTAF approach gives you a much more stable training path than before <ref:2605.11558#pg4>.

Tom: It seems like the paper concludes that HTAF is a powerful tool because it provides stability for binary representations without sacrificing performance when you use it in models like ICBMs <ref:2605.11558#pg32>.

Jane: So, to sum up, they’ve given us a smooth activation function that lets us train neural networks with binary concepts reliably and interpret those features <ref:2605.11558#pg34>. Now, we have to ask what the limits are for these models when you move beyond standard image datasets.

More episodes

← Home