No Routing Needed Between Capsules

arXiv:2001.09136 · cs.CV, cs.LG · Submitted 2020-01-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "No Routing Needed Between Capsules".

Jane: The gist The use of Homogeneous Vector Capsules (HVCs) allows for capsule networks to perform as well as prior state-of-the-art capsule networks on MNIST with significantly fewer parameters and training…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re looking at "No Routing Needed Between Capsules," and this paper starts by pointing out that most capsule network designs rely on traditional matrix multiplication between layers.

Jane: And they say that this matrix multiplication introduces a problem called dimensional entanglement because those dimensions get tangled up.

Lu: The authors propose using Homogeneous Vector Capsules, or HVCs, which use element-wise multiplication instead of matrix multiplication to solve this entanglement issue.

Tom: They argue that by using element-wise multiplication, the capsule dimensions remain unentangled, which is the core thesis of their work.

Jane: They then apply this idea to the highly structured MNIST dataset to produce a direct comparison against established capsule research from Geoffrey Hinton and his team.

Lu: The main claim they show is that this simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST, but with fewer parameters and training epochs.

Tom: Specifically, they show it requires five point five times fewer parameters and four times fewer epochs of training compared to the prior work <ref:2001.09136#pg2>.

Jane: And this is impressive because they also manage to do it without needing a reconstruction sub-network or any routing mechanism at all for the classification task on MNIST.

Lu: The underlying motivation for this paper is exploring alternative ways to achieve multiple paths in a network, like different levels of abstraction or receptive fields, and HVCs seem to address this need without the routing overhead.

Tom: So it matters because it shows that you can get high performance on structured data by changing the fundamental way you multiply things together inside the capsules.

Jane: And they’ve shown this works across several datasets like MNIST, Fashion-MNIST, CIFAR-ten and CIFAR-one hundred which makes the result more general than just being a fluke on one dataset <ref:2001.09136#pg3>.

Lu: It suggests that structural design choices—like branching the network early—can be very effective for getting different views of an image data.

Tom: So what we’re hearing is that HVCs are a viable path to efficient capsule networks by simplifying the mathematical operations involved in layer connections.

Jane: And this opens up a new direction for researchers who want to build capsule models that are more efficient and less reliant on complex routing logic.

Conclusion: Tom: The paper "No Routing Needed Between Capsules" by Adam Byerly et al. is showing that you can achieve top-tier performance on a standard image classification task without the usual computational baggage of routing mechanisms in capsule networks.

Jane: So, to simplify it further, the authors are saying that for structured data like MNIST, you don't need those complicated paths or matrix multiplications to get good results.

Lu: They’ve essentially shown that using element-wise multiplication instead of matrix multiplication is a powerful tool for keeping the capsule dimensions clean and manageable during training.

Tom: And the efficiency numbers they reported—five point five times fewer parameters and four times fewer epochs—are what make this interesting from an engineering standpoint when you compare it to older methods <ref:2001.09136#pg2>.

Jane: It changes things because it means that we might not need to spend as much time designing complex routing logic if we can use these simpler capsule designs instead.

Lu: The implication is that structural choices, like having multiple branches early on, are a better way to get diverse information views than trying to force every signal through a single, complicated routing path.

Tom: So this isn't about inventing a completely new architecture from scratch; it’s about showing how tweaking the math inside existing components can lead to much better performance and efficiency.

Jane: It suggests that for future work, researchers should look at how these HVCs apply to more complex, unstructured data where those entanglement problems might be even worse.

Lu: Exactly. The paper demonstrates that this approach works across a range of datasets, suggesting it’s a solid direction for representation learning research moving forward.

Department of Electronic and Electrical Engineering Brunel University London · Department of Computer Science and Information Systems Bradley University

cs.CV, cs.LG

Submitted: 2020-01-24

Updated: 2021-06-17

Comments: 13 pages, 7 figures, 9 tables

DOI: 10.1016/j.neucom.2021.08.064

Code: https://github.com/AdamByerly/BMCNNwHFCs

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The use of Homogeneous Vector Capsules (HVCs) allows for capsule networks to perform as well as prior state-of-the-art capsule networks on MNIST with significantly fewer parameters and

Key concepts

Homogeneous Vector Capsules (HVCs): HVCs are a capsule network design that uses element-wise multiplication instead of complex matrix multiplication between layers. This approach keeps the dimensions of the capsules unentangled, simplifying the network structure and improving performance compared to traditional methods.
Dimensional Entanglement: In standard capsule networks, matrix multiplication between layers can cause a problem where the dimensions of the capsules become entangled. HVCs are specifically designed to avoid this entanglement by using element-wise operations, which keeps the capsule dimensions independent and manageable.
Branching CNN Architecture: The proposed network uses a convolutional neural network structure with multiple classification branches. These branches operate on different receptive fields of the input image, allowing each branch to learn distinct features and make its own classification prediction simultaneously.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Entanglement — 1× in this paper · explained in 4 episodes

Summary

The gist The use of Homogeneous Vector Capsules (HVCs) allows for capsule networks to perform as well as prior state-of-the-art capsule networks on MNIST with significantly fewer parameters and training epochs, without requiring a routing mechanism

Key Findings and Contributions

The work presents a simple convolutional neural network utilizing HVCs that achieves high accuracy on the MNIST dataset <ref:2001.09136#pg2> The study demonstrates that this design performs as well as the prior best performing capsule network on MNIST using 5.5× fewer parameters, 4× fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism <ref:2001.09136#pg2> This results in an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate) <ref:2001.09136#pg2> A key contribution is the novel method for branching a CNN that allows for multiple effective receptive fields and levels of abstraction where each branch makes it’s own classification prediction <ref:2001.09136#pg3>

Proposed Network Design

The network starts with a conventional CNN following practices like stacked 3 × 3 convolutions, each of which with ReLU [18] activation preceded by batch normalization [19] <ref:2001.09136#pg3> The design intentionally avoids pooling operations because the information it “throws away” <ref:2001.09136#pg3> Dimensionality reduction is handled by choosing not to zero-pad the convolution operations and thus each convolution operation by its nature reduces the dimensionality by 2 in both the horizontal and vertical dimensions <ref:2001.09136#pg4> The architecture branches after the first two sets of three convolutions, in addition to feeding to the subsequent convolution, we also branched off the output to be forwarded on to an additional operation <ref:2001.09136#pg4> This results in three branches in our network with receptive fields of 7 of the original image pixels, 11 of the original image pixels, and 15 of the original image pixels respectively <ref:2001.09136#pg4>

Capsule Construction and Merging

For each branch, filters are transformed into vectors to form capsules, specifically using either XY-Derived Capsules or Z-Derived Capsules <ref:2001.09136#pg5> The process involves element-wise multiplication of these capsules with weight vectors, followed by summing the components to form the second capsule in each pair <ref:2001.09136#pg4> The branch-level logits are formed by reducing the second capsule vector to a single value per class by summing the components of the vector <ref:2001.09136#pg4> To reconcile these branch-level logits, they are stacked into vectors of length 3 and then reduced by summation to form the final set of logits for classification <ref:2001.09136#pg4>

Experimental Setup and Results

The experiments compare different merge strategies, including Not learnable, Random Init., and Ones Init. for branch weights <ref:2001.09136#pg5> The results show that using Z-Derived Capsules was superior to XY-Derived Capsules, as the latter showed only a statistically significant result when branches were allowed to learn their weights <ref:2001.09136#pg6> The ensemble method, which uses the same architecture with different weights, achieved 99.87% accuracy on MNIST <ref:2001.09136#pg7>

Data Augmentation and Ablation

A robust data augmentation strategy was employed for MNIST involving four operations: Rotation, Translation, Width, and Random Erasure <ref:2001.09136#pg6> The translation method used in this work, which involves translating by up to the full margin available in any given direction, was shown to be statistically superior to the standard 2-pixel translation method <ref:2001.09136#pg9> Ablation experiments confirm that HVCs are statistically significantly superior to a fully connected layer for both 1 and 3 branches, and shows that 3 branches are superior to 1 branch for both HVCs and a fully connected layer <ref:2001.09136#pg11> The final design achieved the same level of accuracy as the prior work but with 5.5× fewer parameters, 4× fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism <ref:2001.09136#pg9>

Conclusion

The paper establishes a new state of the art for MNIST with an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate) <ref:2001.09136#pg2> The proposed design and augmentation strategy allow the model to be trained on consumer hardware, enabling training of all 322 trials referenced in the work <ref:2001.09136#pg6> The study concludes by showing that the Z-Derived HVCs and 3 branches achieved a higher mean accuracy with statistical significance across MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets <ref:2001.09136#pg12> The code used for all experiments is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The loss strategy comparison showed no statistically significant difference among the three loss methods <ref:2001.09136#pg11> Finally, using only 2-pixel translation (as in [1]) as the data augmentation strategy allowed for a direct comparison to prior work, yielding the same accuracy with fewer parameters and epochs <ref:2001.09136#pg9> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.09136#pg10> The code used for all experiments and summary level data is publicly available on GitHub at https://github.com/AdamByerly/BMCNNwHFCs <ref:2001.09136#pg13> The paper also demonstrates the ability to interpret capsule dimensions by perturbing individual dimensions of the second set of capsules, revealing meaningful changes in reconstructed images <ref:2001.

Improvements for AI systems

The HVC Architecture for Dimensional Entanglement Resolution: Improving Capsule Networks by Replacing Matrix Multiplication with Element-wise Multiplication (EWM). By using Homogeneous Vector Capsules (HVCs), which use element-wise multiplication rather than matrix multiplication, the dimensions of the capsules remain unentangled. This allows the network to handle the capsule dimensional entanglement that the matrix multiplication introduces, leading to simpler, less computationally expensive designs requiring 5.5× fewer parameters and 4× fewer training epochs.
Multi-Branch Receptive Field Capture: Implementing Branching for Multiple Effective Receptive Fields and Abstraction Levels. We present a novel method for branching a CNN that allows for multiple effective receptive fields and levels of abstraction where each branch makes it’s own classification prediction. This enables the network to capture different levels of abstraction, as opposed to prior methods that rely solely on single paths.
Adaptive Classification Branch Merging: Comparing Static Weighting versus Learnable Weights. In this work we include and compare the results of both statically weighting the classification branches and learning the weights of the classification branches via backpropagation. This allows for an investigation into whether the relative importance of each output is determined statically or learned dynamically during training.
Domain-Specific Data Augmentation: Creating Robust Models with Targeted Image Transformations. By modern standards, in terms of dataset size, MNIST has a relatively low number of training images, and the work applies transformations informed specifically by the domain of the data. This includes specific operations like rotation, translation (with label-preserving constraints), width adjustment, and random erasure to expose the model to a greater variety of (simulated) connectedness within the strokes that make up the digits.
Model Complexity Reduction: Achieving State-of-the-Art Performance with Minimal Parameters. "We show that a simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST using 5.5× fewer parameters, 4× fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism." This demonstrates that high accuracy (99.87% ensemble) can be achieved with a significantly leaner architecture compared to previous designs.