An all-transistor physical-learning transformer can be trained to perform in-context learning

arXiv:2610.12571 · cs.LG, cond-mat.soft · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "An all-transistor physical-learning transformer can be trained to perform in-context learning".

Jane: The gist An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable training with cross-entropy…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So let’s talk more about this paper, "An all-transistor physical-learning transformer can be trained to perform in-context learning." We already touched on how they claim that standard physical learning systems struggle with associative tasks and how these struggle compared to transformers > Page one - <ref:2610.12571#pg3>.

Jane: Right. The main thesis seems to be that they are going beyond just equilibrium propagation by explicitly restoring the bidirectional information flow in a double or dyadic circuit, which is necessary because ideal voltage multipliers are non-reciprocal > Page two - <ref:2610.12571#pg3>.

Lu: They argue that this restoration allows them to construct an attention mechanism by exploiting the exponential current law of subthreshold MOSFET devices, which is a natural way to replace the dot product with something more analogous to Bahdanau attention > Page two - <ref:2610.12571#pg3>.

Meng: So they're taking an MLP structure and making it work as a transformer by fixing the multiplication problem through this circuit modification > Page two - <ref:2610.12571#pg3>. I wonder if that’s feasible when you have to deal with these very specific transistor dynamics.

Lalam: Lalam confirms that in simulations, this circuit can be trained to perform a simple in-context learning task, even though it uses realistic reverse components which only transmit an approximate gradient > Page one - <ref:2610.12571#pg3>. It shows the system can learn something meaningful despite the imperfections.

Tom: That’s cool because it means we don't need perfect physics just to start training on these complex tasks; we just need a nudge in the right direction > Page one - <ref:2610.12571#pg3>. But they stress that this specific training requires cross-entropy loss, not mean squared error > Page one - <ref:2610.12571#pg3>.

Jane: So, if you’re listening who doesn’t read this paper, what does it really mean? It means AI can learn tasks that require associative recall and context in a way traditional neural networks couldn't achieve > Page one - <ref:2610.12571#pg3>.

Lu: It suggests that the architecture itself has potential when implemented in an analog physical system, but you just have to build the right circuit structure to unlock that potential > Page two - <ref:2610.12571#pg3>. It’s not just about brute force computation.

Meng: From a production side, I have to wonder about the cost of building those custom circuit elements—they sound like they add complexity that might make them too expensive for real-world deployment > Page one - <ref:2610.12571#pg3>. What breaks at scale is often the bespoke parts.

Lalam: Lalam has seen the literature, and they connect this to how it changes what we need in terms of hardware design for AI > Page two - <ref:2610.12571#pg3>. It’s about designing circuits that inherently support learning rather than just computation > Page two - <ref:2610.12571#pg3>.

Jane: So, if you’re listening who doesn’t read this paper, the big idea is that analog systems can be trained to do complex things without needing perfect physics all the time > Page one - <ref:2610.12571#pg3>.

Conclusion: Tom: So to wrap up, this paper on "An all-transistor physical-learning transformer can be trained to perform in-context learning" shows that by fixing the flow issue in a double or dyadic circuit, you can build a transformer architecture that learns associative recall tasks > Page two - <ref:2610.12571#pg3>.

Jane: And the authors are Michael C. Abbott and Benjamin B. Machta, and what this means is that we might see physical systems move from just being computation engines to being learning agents capable of understanding context > Page one - <ref:2610.12571#pg3>.

Lu: It pushes the idea that the architecture matters more than just the material it's made of; it’s about how you structure those interactions to enable complex learning > Page two - <ref:2610.12571#pg3>. If this principle is true, we should be designing circuits to support relational reasoning from the start.

Meng: I still see a lot of work left in terms of making this practical and scalable enough to actually ship something that doesn't just sit in a lab > Page one - <ref:2610.12571#pg3>. The complexity of the custom circuit elements is still a big deployment concern for me.

Lalam: Lalam is confident that the core mechanism works, but they show that if you want to do in-context learning, you need to train those specific key and query resistors > Page three - <ref:2610.12571#pg3>. That’s a necessary condition for achieving that specific capability.

Tom: So the paper on "An all-transistor physical-learning transformer can be trained to perform in-context learning" is about taking the architecture and showing how to make it work physically > Page two - <ref:2610.12571#pg3>. It’s about proving that this approach can handle associative recall tasks > Page one - <ref:2610.12571#pg3>.

Jane: And for someone listening, it means we are looking at a future where AI systems are fundamentally shaped by the physical laws they obey, not just digital software on top of them > Page one - <ref:2610.12571#pg3>. We're seeing how to connect the dots between physics and machine learning in a way that enables context awareness > Page one - <ref:2610.12571#pg3>.

Michael C. Abbott, Benjamin B. Machta

Yale Physics & QBio

cs.LG, cond-mat.soft

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 18 pages, 11 figures, plus appendices

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 65/100

The gist: The gist An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable

Key concepts

Physical Learning Paradigm: This is a machine learning method where an analog physical system handles both the forward calculation and back-propagation of errors. The circuit settles into a steady state, and error signals are injected to find a new state that minimizes energy (power dissipation). This allows for precise gradient reconstruction.
Restoring Bidirectional Flow: Ideal circuits often block error signals from flowing backward. To fix this, the authors add extra elements to a double or dyadic circuit. This setup uses the mean of node voltages for forward computation and the difference for the error signal, mimicking how automatic differentiation works in complex systems.
Cross-Entropy Loss Training: Standard training often uses mean squared error, but this system requires cross-entropy loss to succeed in learning. The authors create a novel floating circuit element based on the MOSFET sub-circuit used for softmax to generate the specific nudges needed for this loss function during training.

Terminology

In-Context Learning — 1× in this paper · explained in 7 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable training with cross-entropy loss, allowing it to learn associative recall tasks that are difficult for standard multi-layer perceptrons.

Physical Learning Paradigm

Physical learning is a machine-learning paradigm where an analogue physical system performs both the forward computation and the back-propagation of an error signal to allow training The simplest equilibrium propagation case involves an electrical circuit settling to a free steady state, and during training, the error signal is current injected into the output nodes causing the system to find a new nudged steady state. This scheme works because the circuit’s equilibrium states minimize an energy function, namely the power dissipated, subject to boundary conditions at the input and during training at the output. The contrast between free and nudged currents allows us to precisely reconstruct the gradient of the loss function with respect to this one conductance, enabling adjustment of resistors for gradient descent.

Restoring Bidirectional Flow

The paper addresses the limitation where ideal voltage multipliers are non-reciprocal, completely blocking error signal propagation back to earlier circuit elements. To restore this bidirectional flow, the authors introduce extra circuit elements into a double or dyadic circuit. In this setup, the mean of each pair of node voltages is the forward computation, while the difference is the error signal. This analogy to automatic differentiation works because it mirrors how a chain of functions has pullbacks for error signals.

Transformer-like Architecture

The authors construct an architecture exploiting the exponential nonlinearity of MOSFET devices in a natural way, replacing the dot product with a bump circuit analogous to Bahdanau attention. They implement softmax by sharing a fixed tail current among outputs controlled by input gate voltages. The learned parameters are all adjustable resistors, similar to earlier MLP circuits. The final model is a sequence model where the core operation is dot product self-attention.

Training and Optimization

The training scheme requires cross-entropy loss (not mean squared error) to succeed, and nudges appropriate for this loss are generated using a novel floating circuit element based on the same MOSFET sub-circuit used for softmax. The resistors are trained using the Lion optimiser, which stores local memory of past gradient estimates, obeying the locality of physical learning. Specifically, Adam and Lion store some memory of past updates always for just one parameter.

In-Context Learning Task

The model is tested on a simple in-context learning (ICL) task where the input is a string of characters. The task involves taking the last character in the input and comparing it to all earlier ones to retrieve part of the input near to that match, which is fundamentally what learned induction heads in a transformer do. This ICL task serves as a test that the model truly has new capabilities, as it is very difficult for an MLP to learn.

Conclusion and Conjecture

The paper introduces a conjecture that no physical learning system with pairwise reciprocal interactions can efficiently learn ICL tasks. The double circuit is larger by an O(1) factor, rather than growing with the number of parameters, similar to reverse-mode automatic differentiation. The model shows perfect memorisation of the data with K = 10 classes but cannot learn to perform ICL without training the key and query resistors.

How it works

The double circuit uses two copies of the MLP-like circuit, where the free state and nudge are defined as the mean and difference of these two copies. The gradient is read from the difference in how the two copies behave. The loss function for cross-entropy can be implemented as a floating circuit element exploiting the exponential current law of MOSFETs to implement softmax.

Circuits Adapted for Physical Learning

The paper details several circuit blocks, including Delbrück’s bump circuit [7] used to replace the dot product with a similarity score as a current. The attention mechanism is implemented by translating these pieces to active circuit elements and constructing a double circuit with approximate reverse elements.

Minimal In-Context Learning Task

The ICL task involves input matrices where each character of the string is a 20-dimensional embedding vector, and the desired output is y target = [0, 1, 0, 0, 0, 0], representing “2”. The task can be learned by a one-layer transformer which must pay attention to time t = 1 where the vector representing the query “h” appears.

Non-Reciprocal Interactions

The paper shows that non-reciprocal circuits cannot generally be trained by blindly applying equilibrium propagation, but a larger circuit always exists with the same forward properties and explicit reverse propagation. The dyadic construction of [5] is similar to [4], using +βL(z)+βL(z′) in place of 2βL(z+z′)/2 which produces deviations at O(β 2). The final physical solution there turns off the difference, z − z′ → 0, while [4] keeps it.

Ideal Sub-threshold MOSFET

The ideal device is a 3-terminal component with a current law id→s = I0e(vg−Vth)/nVT h e −vs/VT − e −vd/VT. The dyadic circuit component necessary for a MOSFET to pass error signal backwards is quite complicated, but crude approximations are sufficient for learning.

Adaptive, Local Optimisation Rules

The Lion optimiser stores one number per parameter and the update rule is θt+1 = θt − η sign(ψt+1). This optimisation rule could plausibly be implemented as an RC circuit. The local circuit for Lion would instead have a capacitor which stores the memory ψab, with RC time about 100 training examples.

Transitions between ICL and Memorisation

The in-context learning task is adapted from Reddy [13, 41], who showed a phase transition between generalisation and memorisation, i.e., between ICL and “in-weights learning”. The circuit described in section 3 is a one-layer model capable of learning either strategy. The case K = 10 encourages memorisation, while K = 100 should encourage ICL.

E Nonlinear Circuit Solver

For the truly nonlinear circuits, a custom Newton-Raphson solver is used because MOSFET circuits are badly conditioned. The solution used in our solver is to change variables and rewrite the circuit equations in terms of (va + v′a)/2 and va − v′a. Ngspice is able to solve the final model produced by training and achieves the same accuracy with the same ICL test data.

The gist

An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable training with cross-entropy loss, allowing it to learn associative recall tasks that are difficult for standard multi-layer perceptrons.

Improvements for AI systems

Bold header: Approximate reverse propagation for non-reciprocal systems

The system can perform training on transformer-like architectures by explicitly restoring bidirectional information flow using a double or dyadic circuit structure, which allows for the physical back-propagation of error signal through arbitrary nonlinear elements.

Bold header: Attention mechanism exploiting exponential current law

The model can implement transformer-like behavior by replacing the dot product with a bump circuit [7] analogous to Bahdanau attention [8], leveraging the exponential nonlinearity of MOSFET devices in a natural way.

Bold header: Local, adaptive optimization rules

The training process can utilize optimizers like Lion [11], which store some memory of past updates, always for just one parameter, obeying the locality of physical learning and being implementable as an RC circuit.

Bold header: Learning in-context generalization (ICL)

The system can perform in-context learning (ICL) task[s], such as the associative recall task described, demonstrating that it can learn to compare input characters and retrieve relevant context based on learned rules.

Bold header: Enhanced expressivity beyond MLPs

The architecture is capable of learning tasks that are very difficult for an MLP to learn, suggesting a new capability in handling associative recall and generalization compared to standard multi-layer perceptrons.