CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics

arXiv:2607.23258 · cs.AI, cs.LG · Submitted 2026-07-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics".

Jane: The gist The CAPT model introduces a continuous autoregressive transformer for calcium population dynamics that enables cross-dataset and cross-species transfer for calcium population dynamics How it works CAPT models continuous…

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we just covered the high-level idea of CAPT: this Continuous Autoregressive Population Transformer for calcium population dynamics. To summarize what they are actually proposing, they’re tackling the problem that current models are too tailored to single datasets and want to see if we can build foundation-style models for neural population dynamics.

Tom: They claim that by pretraining on a large-scale mouse dataset, you can learn general representations applicable to diverse tasks like forecasting and decoding behavior. But the real challenge they identify is generalization beyond the data distribution they were trained on because datasets vary a lot in species, brain regions, imaging conditions, sampling rates, neuron counts and experimental paradigms.

Lu: They propose CAPT to solve this by modeling continuous calcium traces directly through a continuous patch tokenization strategy and training it autoregressively with a mean squared error objective. This sets up the end-to-end training process that allows for adaptation to diverse downstream tasks.

Meng: So, the core mechanism is this continuous patching and the autoregressive training loop, right? Can you tell me more about how they handle making sure the predictions are accurate during that autoregressive prediction phase?

Tom: They define their autoregressive objective as a mean squared error over predicted future patches from past activity. To make it more robust to imperfect histories, they apply two lightweight corruptions to the input history while keeping the prediction targets unchanged.

Jane: That adds a layer of robustness, which is important when you’re dealing with continuous data where the history might be noisy or incomplete in a real lab setting. So, it's not just learning patterns; it's learning how to predict sequences reliably even when the input sequence isn't perfect.

Lu: They also looked at related work, specifically classical methods like latent dynamical systems and state-space models for neural forecasting, and then they brought in Transformer architectures to leverage foundation-style modeling of invasive neural recordings.

Tom: And their evaluation was quite broad; they tested this transferability across eight independent datasets: three mouse datasets, three larval zebrafish datasets, and two C. elegans datasets. This is a pretty wide range of biological systems for them to test against.

Jane: It’s impressive that they didn't just test the transfer on a single new dataset, but across different species and labs to prove the model isn't just memorizing mouse-specific features or lab-specific noise. What kind of performance gains did they see when moving between these groups?

Tom: In all transfer experiments, CAPT consistently outperformed specialized and general-purpose baselines like POYO+, POCO, and CalM. And across both the pretraining and transfer datasets, CAPT shows significantly better performance in most cases.

Lu: The two-step protocol they used separates neural population adaptation from task adaptation by only updating new embeddings during forecasting transfer or only updating the decoder during behavior decoding. This directly tests if a single continuous autoregressive backbone can provide those reusable representations across datasets, experimental paradigms, and species.

Meng: If it works like that, it means the backbone is really doing the heavy lifting of learning the general representation for everyone else. But what about deployment? How much of this complex transformer architecture are we talking about needing to run in a real-time system?

Tom: It's an autoregressive model, so inference involves sequentially predicting patches. The complexity is managed by the Transformer structure itself, but they are focused on the adaptation modules being lightweight enough to update efficiently.

Jane: So, if you’re listening and you’re thinking about applying this to a new species or lab setup, what’s the main promise of CAPT that makes it worth paying attention to?

Lu: The promise is that continuous autoregressive modeling provides an effective framework for learning calcium population representations that transfer across datasets and experimental paradigms, which is something we haven't seen consistently before.

Meng: It sounds like a powerful tool, but I still wonder about the training data requirements. They pretrained it on a large-scale mouse dataset; how much more data do you think they need to train it on if they wanted to generalize even further across different phyla?

Tom: The current finding is that freezing the backbone and only updating embeddings works well, suggesting we don't need massive retraining for every new target task. But as Lu mentioned, the limitation is that CAPT is currently pretrained on a single large-scale mouse calcium imaging dataset.

Jane: So, it’s a strong proof of concept for transferability across species and labs, but it’s still rooted in that initial mouse training. That sets us up perfectly for talking about how this impacts the broader field of AI modeling for biological data.

Conclusion: Tom: So wrapping up the discussion on CAPT: its full title is A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics. It’s about moving away from models that are stuck in one experimental setting.

Jane: Essentially, the authors are showing that a single pretrained model can learn a set of neural population representations that transfer across different species, different labs, and even different experimental paradigms when adapted correctly.

Lu: The main implication is that we can build foundation-style models for neural population dynamics by leveraging large-scale recordings to learn general representations applicable to diverse tasks. This moves us past just solving one specific forecasting problem in one specific animal.

Meng: For deployment, this means we have a starting point—a pretrained backbone—that we can fine-tune with minimal data for new biological targets, which is a huge win for researchers who need to apply these models quickly to novel systems.

Tom: I think the thing that stands out is the success of separating neural population adaptation from task adaptation, which directly validates this idea of learning reusable representations. It proves that the continuous autoregressive backbone isn't just fitting noise; it’s learning structure.

Jane: And alongside those predictive wins, they found that these CAPT embeddings capture shared functional space across datasets and actually encode cell-identity related structure in C. elegans data, which is a deep biological finding for us.

Lu: That shared functional space idea means the model is learning something fundamental about how neurons interact or are organized regardless of the specific species it was trained on.

Tom: So, this work suggests that continuous autoregressive modeling is an effective way to learn representations that generalize across datasets and species for calcium imaging data. It’s a simple framework to start with.

Jane: It gives us a solid foundation for how we can approach building AI models that can actually handle the massive heterogeneity of biological data out there.

Tsinghua University

cs.AI, cs.LG

Submitted: 2026-07-25

Updated: 2026-10-18

Code: https://github.com/TSuXinH/CAPT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The CAPT model introduces a continuous autoregressive transformer for calcium population dynamics that enables cross-dataset and cross-species transfer for calcium population dynamics How it

Key concepts

Continuous Patch Tokenization: This technique breaks down long calcium traces into fixed-length temporal segments or 'patches'. These patches are then projected into the model's space using learned embeddings (for neurons and sessions). This allows the transformer to process the continuous activity stream sequentially, treating different time segments as discrete tokens.
Autoregressive Training: The model is trained to predict future calcium activity based on past activity. It uses a mean squared error objective to minimize the difference between its predictions and the actual future traces. This self-supervised method forces the transformer to learn the underlying temporal dependencies and dynamics of neural population changes.
Transfer Learning Protocol: CAPT separates adaptation into two stages: forecasting (for predictive tasks) and behavior decoding (for behavioral tasks). During transfer, only small modules like neuron or session embeddings are updated, while the main backbone is frozen. This proves that a single representation can be reused effectively across different species or datasets.
Shared Functional Space: Analysis shows that the embeddings generated by CAPT for different species and datasets occupy a similar functional space. This means the model learns representations where neurons with similar roles or identities are clustered together, regardless of whether they are from a mouse or a worm.

Terminology

Adaptation — 2× in this paper · explained in 1 episode
Trace — 2× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Tokenization — 1× in this paper · explained in 2 episodes
Transferability — 1× in this paper · explained in 5 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The CAPT model introduces a continuous autoregressive transformer for calcium population dynamics that enables cross-dataset and cross-species transfer for calcium population dynamics

How it works

CAPT models continuous neural activity directly through a continuous patch tokenization strategy and is trained autoregressively with a mean squared error objective, enabling end-to-end pretraining and adaptation to diverse downstream tasks. The model first pretrain CAPT on a large-scale mouse calcium imaging dataset, and then it tests its transferability across independent mouse, larval zebrafish, and C. elegans datasets collected by different laboratories. In these transfer settings, the pretrained backbone is frozen and only adaptation modules, i.e., neuron and session embeddings or task-specific heads, are updated.

The continuous patch tokenization involves partitioning each single-neuron trace into non-overlapping temporal patches of length L, projecting these patches into the model space with neuron and session embeddings, and modeling them autoregressively by a population Transformer. The Dual-axis Transformer (DAT) backbone factorizes population modeling into two complementary directions: causal temporal attention models the history of each neuron, while neuron-axis attention models populationlevel interactions at each patch position. During pretraining, CAPT is pretrained to predict future continuous calcium activity from past activity, and the autoregressive objective is defined as a mean squared error over predicted future patches. To improve robustness to imperfect autoregressive histories, two lightweight corruptions are applied to the input history while keeping the prediction targets unchanged.

Transferability and Performance

CAPT is evaluated on eight independent datasets beyond the pretraining data: three mouse datasets, three larval zebrafish datasets, and two C. elegans datasets. In all transfer experiments, CAPT consistently outperforms specialized and general-purpose baselines, including POYO+, POCO, and CalM. Across both pretraining and transfer datasets, CAPT shows significantly better performance in most cases (Table 12). The two-step protocol separates neural population adaptation from task adaptation: forecasting transfer only updates new embeddings, while behavior decoding only updates the decoder. This two-step protocol directly tests whether a single continuous autoregressive backbone can provide reusable representations across datasets, experimental paradigms, and species.

Biologically Meaningful Embeddings

Alongside predictive performance, multimodal analyses using NeuroPAL annotations in C. elegans datasets show that CAPT embeddings form a shared functional space across datasets and capture anatomical cell-identity-related structure. Beyond forecasting and decoding performance, we find that CAPT embeddings recover identity-related structure and support cross-dataset neuron identity classification using only a linear support vector machine (SVM). This suggests that the model learns a shared functional embedding space across datasets and captures anatomical information. CAPT clearly outperforms POCO (Figure 4B, C, Appendix E), showing that CAPT embeddings preserve stronger cell-identity information.

Conclusion and Limitations

CAPT provides a simple framework for reusing a single pretrained model across different neural populations, laboratories, experimental paradigms, and species for calcium imaging datasets. Several limitations remain; (1) CAPT is currently pretrained on a single large-scale mouse calcium imaging dataset. (2) The performance gain under transfer is generally smaller than that on the pretraining dataset, suggesting that the current backbone may still be partially constrained by the source distribution. (3) The methods tested here generally perform better on trial-aligned datasets than on recordings without clear trial structure, indicating that pseudo-trial segmentation may be insufficient for spontaneous or weakly structured neural activity. (4) The mechanism underlying cross-species transfer remains only partially understood.

The gist CAPT models continuous neural activity directly through a continuous patch tokenization strategy and is trained autoregressively with a mean squared error objective, enabling end-to-end pretraining and adaptation to diverse downstream tasks

Improvements for AI systems

Continuous autoregressive modeling for calcium traces allows CAPT to learn representations that transfer beyond the original pretraining distribution, enabling end-to-end pretraining and adaptation to diverse downstream tasks. This means an AI system can be pretrained on a large mouse dataset and then immediately applied to predict calcium activity in new, unseen datasets or species without extensive task-specific retraining.
Cross-dataset and cross-species transfer capability allows the model backbone to be frozen during adaptation, meaning only adaptation modules, i.e., neuron and session embeddings or task-specific heads, are updated. This enables a single foundation model to generalize across datasets (mice, zebrafish, C. elegans) and experimental paradigms (Y-maze vs. Copper boundary), significantly reducing the need for specialized models for each new organism or lab setting.
Biologically meaningful functional embeddings enable multimodal analysis by showing that CAPT embeddings form a shared functional space across datasets and capture anatomical cell-identity-related structure. This allows researchers to use the learned embedding space, linked with NeuroPAL annotations in C. elegans, to perform cell-identity classification across different species, effectively connecting functional calcium activity to biological cell identity.
Superior performance on diverse tasks means CAPT can achieve superior performance on available behavioral variables even when compared against specialized baselines like POYO+ and CalM in challenging cross-species settings. This capability allows the AI system to perform both neural population forecasting and behavior decoding with high accuracy across heterogeneous environments, such as predicting velocity from larval zebrafish activity or head angle from C. elegans activity.