Disentanglement with Biological Constraints: A Theory of Functional Cell Types

The gist: Simple biological constraints like nonnegativity and energy efficiency lead to disentanglement, which explains why single neurons in the brain often represent single human-interpretable factors.

How it works

1. The theory proves that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity and weights, promote such sought after disentangled representations by enforcing neurons to become selective for single factors of task variation.

2. In linear networks, the theorem shows that under the constraints of nonnegativity and constant population variance, the representations that achieve minimal activity energy also exhibit disentanglement, meaning every neuron is selective for at most one task parameter.

3. The intuition behind this proof is illustrated in Figure 1, where minimizing activity energy subject to nonnegativity and variance constraints requires the smallest possible bias for each neuron, which occurs when each neuron is selective for a single task factor.

4. In more general settings, Theorem 2 demonstrates that any linear, nonnegative, optimally energetically efficient, generative neural representation that accurately predicts entangled observations will possess single neurons that are selective for individual task factors.

Disentanglement in Machines

**- Disentanglement in supervised shallow neural networks demonstrates that with biological constraints, networks learn distinct sub-networks for each task factor when trained on linear data. This effect is driven by nonnegativity rather than sparsity, as ReLUs promote nonnegativity not because they induce sparsity but because they enforce nonnegativity. The deep networks show that early layers are mixed-selective, while later layers are disentangled, suggesting constraints encourage linear relationships to task factors in later layers. In unsupervised learning with autoencoders on linearly mixed data, the model recovers the independent task factors in individual neural subspaces when all constraints are present. The results on standard benchmarks with VAEs show that our constraints lead to disentanglement, often achieving competitive scores even though they do not explicitly ask for a factorised aggregate posterior. In the case of categorical distributions, the proof shows that minimizing expected activity energy requires each neuron to represent at most a single category. The simulation results for this task show that only our constraints, and a sparsity inducing loss, achieve disentanglement of categories. In the context of pattern forming dynamics, the structural loss ensures representations understand structure or rules in the world or behavior through homeomorphism. The constraints also explain why grid cells and object-vector cells emerge as distinct functional cell types when tasks are factorised into independent factors. The warping of grid fields towards rewards is explained as a consequence of space becoming entangled with objects or rewards in behavioural tasks. These constraints provide a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors. The work suggests that when mappings are nonlinear, mixed-selectivity exists but gradually de-mixes as layers approach the output or latent layers. When tasks are factorised into independent factors, biological constraints cause complete disentanglement of the networks if the transformations are linear. The work bridges the gap between single neuron and population responses by showing that representations should be encoded in separate neural populations when task variables correspond to independent factors. When combinatorial generalisation is required, the factored solution may be preferred over mixed-selectivity. The paper suggests that for small values of βw, disentanglement is preferred even for arbitrary D. The work offers a program to understand how brain representations structure themselves by telling us when neural circuits systems should represent different factors in different neurons. The paper suggests that the same principles of nonnegativity, minimizing neural activity, and representing structure will be essential components obtaining this more general understanding. The work demonstrates the utility of neurobiological considerations in designing machine learning algorithms. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that the same principles of nonnegativity, minimizing neural activity, and representing structure will be essential components obtaining this more general understanding. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that the same principles of nonnegativity, minimizing neural activity, and representing structure will be essential components obtaining this more general understanding. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations. The work suggests that for small values of βw disentanglement is preferred even for arbitrary D. The work offers a unified research program that more deeply connects the neuroscience and machine learning communities to help in their combined quest to both understand and learn neural representations.

1. Biologically constrained linear models can be used in machine learning to ensure that single neurons represent single factors of task variation, leading to interpretable representations. This allows for building artificial neural networks where individual neurons map directly to human-interpretable concepts like spatial location or object presence.

2. Deep neural networks trained with nonnegativity and energy efficiency constraints will exhibit disentangled representations in both linear and nonlinear tasks, even when the input data is entangled. This capability is crucial for understanding how complex brain areas partition their cells into distinct functional cell types, such as grid cells and object-vector cells.

3. Variational Autoencoders incorporating biological constraints can recover independent task factors from latent spaces when trained on factorized data, which helps in creating latent representations that are not just decorrelated but also genuinely factorized into single dimensions.

4. In robotics or reinforcement learning tasks involving spatial navigation and object interaction, a representation can be structured to predict location, object presence, and optimal action simultaneously using path integration constraints. This enables the system to learn representations that encode both allocentric space and object-centric coordinates in separate modules when the task is factorized.

5. A new metric called mutual information ratio (MIR) can be used to evaluate disentanglement beyond simple mutual information gap metrics, scoring high when single neurons show high preference for a single ground truth factor. This provides a more nuanced way to quantify the desired level of neural representation structure in complex models.

Neurons in the brain are often finely tuned for specific task variables. Moreover, such disentangled representations are highly sought after in machine learning. Here we mathematically prove that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity and weights, promote such sought after disentangled representations by enforcing neurons to become selective for single factors of task variation. We demonstrate these constraints lead to disentanglement in a variety of tasks and architectures, including variational autoencoders. We also use this theory to explain why the brain partitions its cells into distinct cell types such as grid and object-vector cells, and also explain when the brain instead entangles representations in response to entangled task factors. Overall, this work provides a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors, and steps towards an understanding task structure shapes the structure of brain representation.