A Composite Activation Function for Learning Stable Binary Representations

arXiv:2605.11558 · cs.LG, stat.ML · Submitted 2026-05-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "A Composite Activation Function for Learning Stable Binary Representations".

Jane: Here's the gist of the paper. The gist: This paper proposes Heavy-Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function, which enables stable training with standard gradient-based optimization for neural networks with binary representations, and introduces Implicit Concept Bottleneck Models (ICBMs) that leverage HTAF to induce discrete feature representations.

Tom: Heavy-Tailed Activation Function (HTAF) The authors propose Heavy-Tailed Activation Function (HTAF), which is defined as HTAF(xα0, α1):= σ(α1ψ(α0x)) (Page 3).

Jane: This function is constructed as a sigmoid–hyperbolic tangent composite function to approximate the Heaviside function while ensuring stable training with gradient-based optimization (Page 2).

Tom: The key idea behind HTAF is to design an activation function that "preserves a large gradient near zero (to approximate the Heaviside function closely), while allowing the gradient to decay slowly for large-magnitude inputs (to avoid the gradient vanishing problem)" (Page 2).

Paper summary: Tom: So, the paper "A Composite Activation Function for Learning Stable Binary Representations" introduces HTAF as a smooth approximation to the Heaviside function, constructed from a sigmoid hyperbolic tangent composite function <ref:2605.11558#pg3>.

Jane: And what it does is enable stable training with gradient-based optimization for neural networks that use binary representations, which was previously very challenging because the Heaviside function isn't differentiable <ref:2605.11558#pg2>.

Lu: The key idea they are pushing is designing an activation function that keeps a large gradient near zero inputs while allowing the gradient to decay slower in the tail regions, which lets them control the rate of gradient decay by adjusting alpha zero and alpha one <ref:2605.11558#pg2>.

Meng: This means for deployment, we have a more stable training method for binary models that doesn't require surrogate gradients like STE <ref:2605.11558#pg3>.

Tom: And the introduction of Implicit Concept Bottleneck Models, or ICBMs, shows this approach can induce binary latent representations corresponding to implicit concepts while retaining high prediction accuracy <ref:2605.11558#pg3>.

Jane: It’s about taking those efficient binary features and making them interpretable by showing how they relate to learned implicit concepts, which is a major step for AI understanding.

Lu: The paper concludes that HTAF is a novel activation function that stably approximates the Heaviside function under gradient-based optimization by preserving gradient magnitude <ref:2605.11558#pg5>.

Meng: It provides an effective way to train DNNs with the Heaviside activation function using a standard gradient descent algorithm without surrogate gradients, while significantly improving training stability and prediction performance <ref:2605.11558#pg3>.

Tom: That’s the core idea: HTAF is a novel activation function that stably approximates the Heaviside function under gradient-based optimization by preserving gradient magnitude <ref:2605.11558#pg5>.

Jane: And the introduction of ICBMs demonstrates that this approach can induce binary latent representations corresponding to implicit concepts while retaining high prediction accuracy <ref:2605.11558#pg3>.

Conclusion: Tom: So, we’re wrapping up this deep dive on Heavy Tailed Activation Function, or HTAF. This paper is about how they built a smooth way to approximate the Heaviside function for training neural networks with binary representations <ref:2605.11558#pg3>.

Jane: Right, so it’s about taking something that isn't differentiable and making it usable with standard gradient methods. The authors, they built this HTAF as a sigmoid hyperbolic tangent composite function <ref:2605.11558#pg2>.

Lu: The core idea they push is controlling the gradient behavior. They designed it so that near zero inputs, the gradient stays large enough to approximate that sharp Heaviside jump, but in the tail regions, it decays slower <ref:2605.11558#pg2>.

Meng: From a deployment standpoint, what matters is stability at scale. If you’re running binary models on production hardware, you need training that doesn't blow up because of tiny numerical issues <ref:2605.11558#pg3>.

Tom: Exactly. And they show that by tuning those two parameters, alpha zero and alpha one, you can manage that gradient decay in the tail regions without messing up the gradient near zero inputs <ref:2605.11558#pg2>.

Jane: That means for networks using binary representations—like spiking or deep Heaviside neural networks—you can train them stably using standard gradient descent, not needing those extra tricks like surrogate gradients <ref:2605.11558#pg3>.

Lu: And they show this works across several network types, including spiking neural networks and binary neural networks <ref:2605.11558#pg3>. It's a solid theoretical foundation for working with those discrete representations.

Meng: I’m curious about the application side. They introduce Implicit Concept Bottleneck Models, or ICBMs, which use this HTAF to induce discrete feature representations <ref:2605.11558#pg2>.

Tom: That’s where it gets interesting for interpretation. ICBMs use HTAF in the final layer during training to find these binary concepts, and then at inference time, they switch back to the Heaviside function for truly discrete outputs <ref:2605.11558#pg2>.

Jane: So it’s not just making them train better; it’s about making those features inherently interpretable by linking them to learned concepts <ref:2605.11558#pg7>.

Lu: They claim this allows the AI to learn these implicit concepts directly from the data, instead of asking a human to label things first <ref:2605.11558#pg7>.

Meng: And in their experiments, they show that ICBMs achieve prediction performance that’s comparable to or even better than standard image models on datasets like CIFAR-ten and CIFAR-one hundred <ref:2605.11558#pg32>. That's a big win for practical deployment.

Tom: So the authors are basically saying they found a way to get the benefits of binary representations—efficiency and interpretability—while keeping the performance high with this HTAF activation <ref:2605.11558#pg4>.

Jane: It’s a lot of moving parts, though. They do mention that tuning alpha zero and alpha one helps with vanishing gradients, but they also flag that you need to be careful; if you tune them too much, the numerical stability can actually get worse <ref:2605.11558#pg5>.

Lu: That’s a fair caveat. They show the limit of the derivative is mostly governed by alpha zero, but at the origin, it depends on both alpha zero and alpha one <ref:2605.11558#pg32>. It’s a bit nuanced there.

Meng: For me, what I'm hearing is that if you want to build efficient binary models that actually work reliably in the real world, this HTAF approach gives you a much more stable training path than before <ref:2605.11558#pg4>.

Tom: It seems like the paper concludes that HTAF is a powerful tool because it provides stability for binary representations without sacrificing performance when you use it in models like ICBMs <ref:2605.11558#pg32>.

Jane: So, to sum up, they’ve given us a smooth activation function that lets us train neural networks with binary concepts reliably and interpret those features <ref:2605.11558#pg34>. Now, we have to ask what the limits are for these models when you move beyond standard image datasets.

Seokhun Park, Choeun Kim, Kwanho Lee, Sehyun Park, Insung Kong, Yongdai Kim

Seoul National University · University of Twente

cs.LG, stat.ML

Submitted: 2026-05-12

Updated: 2026-10-21

Comments: 32 pages

Code: https://github.com/pytorch/vision

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: This paper proposes Heavy-Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function, which enables stable training with standard gradient-based optimization for

Key concepts

Heavy Tailed Activation Function (HTAF): HTAF is a composite function designed to smoothly approximate the sharp Heaviside step function. It is constructed using a sigmoid and a hyperbolic tangent combination. Its key feature is that it maintains a large gradient near zero for stable training, while its tail region gradient decay can be controlled by parameters.
Gradient Mass Preservation: This property refers to HTAF's ability to keep the gradient magnitude high around zero inputs. This is crucial because standard optimization methods struggle with non-differentiable functions like the Heaviside function, which have zero derivative everywhere except at the step. HTAF ensures a usable gradient exists near this critical point.
Implicit Concept Bottleneck Models (ICBMs): ICBMs are an interpretable model extension that uses HTAF to create discrete feature representations. Instead of relying on pre-defined concepts, ICBMs learn implicit concepts directly from data using HTAF. These learned features can then be interpreted by external models like Large Language Models, yielding class-specific binary representations.
Binary Neural Networks (BNNs): BNNs are neural networks trained to produce binary outputs instead of continuous values. Training these networks is challenging because the Heaviside activation function is not differentiable. HTAF provides a stable solution by offering a smooth, differentiable approximation that allows for effective gradient-based optimization during the training process.

Terminology

Activation Function — 3× in this paper · explained in 1 episode
Heavy Tailed Activation Function — 1× in this paper
Interpretability — 1× in this paper · explained in 15 episodes
Spiking neural networks — 1× in this paper · explained in 2 episodes

Summary

The gist: This paper proposes Heavy-Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function, which enables stable training with standard gradient-based optimization for neural networks with binary representations, and introduces Implicit Concept Bottleneck Models (ICBMs) that leverage HTAF to induce discrete feature representations.

Heavy-Tailed Activation Function (HTAF)

The authors propose Heavy-Tailed Activation Function (HTAF), which is defined as HTAF(xα0, α1):= σ(α1ψ(α0x)) (Page 3). This function is constructed as a sigmoid–hyperbolic tangent composite function to approximate the Heaviside function while ensuring stable training with gradient-based optimization (Page 2). The key idea behind HTAF is to design an activation function that preserves a large gradient near zero (to approximate the Heaviside function closely), while allowing the gradient to decay slowly for large-magnitude inputs (to avoid the gradient vanishing problem) (Page 2). HTAF has two parameters, α0 and α1, which allow the gradient magnitude in the tail regions to be adjusted without affecting the gradient near zero (Page 2).

Theoretical Properties of HTAF

The authors theoretically show that by appropriately tuning its two parameters, HTAF can control the rate of gradient decay in tail regions while keeping a large gradient near zero, thereby enabling stable learning of deep neural networks with HTAF activation with standard gradient-based optimization (Page 2). Specifically, Theorem 3.2 summarizes this property: the limit of the derivative in the tail region is governed mostly by α0 (Page 5), while the slope at the origin is determined by α0 and α1 (Page 5). The authors also compare HTAF's gradient behavior with Scaled Sigmoid (SS), showing that the gradient of HTAF is much larger than that of SS in the tail regions (Page 5).

Applications in Neural Networks

HTAF enables stable end-to-end optimization for several network types:

Spiking Neural Networks (SNNs, [66, 16, 45]) (Page 2).
Binary Neural Networks (BNNs, [52, 55, 42]) (Page 3).
Deep Heaviside Neural Networks (DHNs, [34, 14]) (Page 3).

As an application of HTAF, the authors propose Implicit Concept Bottleneck Models (ICBMs), which employ HTAF as the activation function in the final layer of the backbone feature extractor during training (Page 2). At inference time, HTAF is replaced by the Heaviside function, yielding exact binary activations and resulting in fully discrete and interpretable features without substantially sacrificing accuracy (Page 2).

Implicit Concept Bottleneck Models (ICBMs)

ICBMs are an extension of Concept Bottleneck Models (CBMs) that leverage HTAF to induce binary latent representations corresponding to implicit concepts (Page 2). The authors propose a new approach to obtaining concepts for CBMs: instead of first asking an AI system to define or annotate concepts, we first learn concepts directly from data and then ask AI to interpret their meanings (Page 7). For implicit concepts to be desirable, they must satisfy properties including Binarity, Class specificity, and Completeness (Page 7). ICBMs are shown to estimate implicit concepts using HTAF while maintaining prediction performance comparable to or better than conventional image models (Page 2).

Experimental Results

Extensive experiments demonstrate the effectiveness of HTAF:

** Deep Neural Networks with the Heaviside activation function**

Table 1 shows that when using HTAF or SS, all Heaviside activation functions are replaced with these functions during training and reverted to the Heaviside function at inference (Page 4). Results indicate that HTAF achieves prediction performance competitive with STE in learning SNNs, while consistently outperforming both STE and SS in BNNs and DHNs (Page 4).

** Implicit Concept Bottleneck Models**

Table 11 shows that ICBMs achieve substantially higher prediction performance compared to baseline methods on CIFAR-10, CIFAR-100, and CUB-200 datasets (Page 32). Furthermore, HTAF exhibits the smallest degradation in prediction performance after inference-time binarization compared to SS and BM (Page 32).

Conclusion

The paper concludes that HTAF is a novel activation function that stably approximates the Heaviside function under gradient-based optimization by preserving gradient magnitude (Page 5). It further shows that HTAF provides an effective way to train DNNs with the Heaviside activation function using a standard gradient descent algorithm without surrogate gradients, while significantly improving training stability and prediction performance (Page 5). The introduction of ICBMs demonstrates that this approach can induce binary latent representations corresponding to implicit concepts while retaining high prediction accuracy (Page 5).

How it works

HTAF is constructed as a sigmoid–hyperbolic tangent composite function: HTAF(xα0, α1):= σ(α1ψ(α0x)) (Page 3).
It maintains a large gradient near zero and allows the gradient to decay slowly in the tail regions by adjusting parameters α0 and α1 (Page 2).
For DNNs with Heaviside activation functions, HTAF is used during training, and then replaced by the Heaviside function at inference time (Page 6).
ICBMs use HTAF in the final layer of the feature extractor to estimate implicit concepts, which are then interpreted using a Large Language Model (LLM) (Page 7).

Key Findings

HTAF enables stable learning of DNNs with Heaviside activation functions using standard gradient descent algorithm without surrogate gradients (Page 5).
ICBMs utilize HTAF to induce binary latent representations that are class-specific, and the learned implicit concepts can be interpreted by an LLM (Page 9).
HTAF maintains competitive or improved prediction performance across various architectures and datasets in ICBMs compared to conventional image models (Page 2).
Tuning α0 and α1 can alleviate gradient vanishing, but excessive values may cause numerical instability, suggesting a careful selection is required (Page 5).

Future Work

The authors note that addressing the complete elimination of gradient vanishing remains an important direction for future research, as well as investigating principled methods for determining the number of concepts in ICBMs and the theoretical properties of DNNs employing HTAF (Page 10). The paper also suggests that tuning α0 can partially mitigate gradient vanishing, but does not provide a complete solution (Page 25).

Impact Statement

This research aims to advance the field of Machine Learning by proposing HTAF and ICBMs, which offer stable training and interpretable binary representations for neural networks (Page 15). The work has broader societal implications, though specific consequences are not identified here (Page 15).

Proof of Proposition 3.1

The proof for Proposition 3.1 shows that HTAF uniformly approximates the Heaviside function outside an arbitrarily small neighborhood of the origin under specific conditions on α0 and α1 (Page 4). The proof involves finding ranges of parameters satisfying inequalities derived from the definition of σ(z) and ψ(x) (Page 15).

Proof of Theorem 3.2

The proof for Theorem 3.2 confirms that the gradient in the tail region is governed by α0, and the slope at the origin is determined by α0 and α1 (Page 5). The proof involves direct calculation of the derivative of HTAF with respect to x (Page 15).

Approximation of a DNN with Heaviside Activation Function

Theorem C.1 proves that a Deep Heaviside Neural Network can be approximated by a DNN with HTAF activation as long as the Heaviside function is well approximated by HTAF, showing that EX[f(X) − g(X)] ≤ n1∥W1∥∞(1 + C)ϵ (Page 20).

Statistical Convergence Rate

Theorem B.3 demonstrates that DNN estimators with HTAF achieve the minimax optimal rate over the Hölder class, and the dependence on α0 and α1 appears only through logarithmic factors (Page 19).

Comparison to ANN-to-SNN method

Table 5 compares prediction accuracy and inference time between ANN-to-SNN conversion and SNN with HTAF, showing that our method maintains competitive accuracy with significantly fewer time steps (e.g., T=4), resulting in substantially efficiency inference compared to the ANN-to-SNN method (Page 23).

Effect of post-binarization on model prediction performance

Table 10 shows that HTAF exhibits the smallest degradation in prediction performance after inference-time binarization compared to SS and BM (Page 32).

Analysis of representations by HTAF

Figure 12 shows that when using HTAF, most of the feature extractor outputs are concentrated near zero or one, whereas this behavior is not observed when using ReLU (Page 32).

Improvements for AI systems

Replace Heaviside activations in Spiking Neural Networks, Binary Neural Networks, and Deep Heaviside Neural Networks with Heavy-Tailed Activation Function (HTAF) during training to enable stable end-to-end optimization without surrogate gradients for activation functions.
Implement Implicit Concept Bottleneck Models (ICBMs) using HTAF as the activation function in the final layer of the backbone feature extractor during training to induce binary latent representations corresponding to implicit concepts, achieving prediction performance comparable to or better than standard models.
Utilize HTAF for surrogate-free training of neural networks with Heaviside activation functions, allowing the network to be optimized using a standard gradient descent algorithm while preserving learned weights and producing exact discrete activations at inference time.
Design parameter selection strategies for HTAF by considering the condition α0α1 = 22 when choosing small α0 and large α1 to maintain a sufficiently large gradient near zero while alleviating gradient vanishing in the tail regions.

Abstract

Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.