A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli

summary

Video file (mp4)

The gist

The gist This sequence-to-sequence Transformer predicts whole-brain fMRI responses to naturalistic multimodal movies by autoregressively predicting activity from visual, auditory, and language inputs.

In short

This research proposes a sequence-to-sequence Transformer to predict whole-brain fMRI responses to naturalistic movies by autoregressively processing visual, auditory, and language inputs. It uses specialized models like VideoMAE and HuBERT for feature extraction and integrates narrative summaries to capture complex temporal dependencies in brain activity.

Key concepts

VideoMAE
This is a pre-trained model used to extract meaningful representations from video frames. It helps the system understand the temporal dynamics of visual motion, allowing it to create a richer understanding of what is happening visually at each point in time.
HuBERT
HuBERT is a self-supervised model applied to audio signals. It learns hierarchical speech representations by predicting masked parts of the audio, capturing both the specific sounds (phonetics) and the emotional tone or rhythm (prosody) of speech.
Sequence-to-Sequence Transformer
This is a type of deep learning architecture designed to map one sequence (the multimodal movie inputs: video, audio, text) to another sequence (the fMRI brain activity). It allows the model to generate the brain response step-by-step based on all the preceding inputs.
Subject Variability Handling
The model uses a shared encoder but has separate decoder heads for each subject. Each subject gets a unique embedding vector that is added to every time step of the decoder, allowing it to adapt its prediction specifically to that individual's brain response patterns.

Terminology used across episodes

This episode discusses

The paper

A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli · Read on arXiv

Qianyi He, Yuan Chang Leong

Data Science Institute, University of Chicago · Department of Psychology, Neuroscience Institute, University of Chicago

Brain encoding models predict time-resolved neural activity from computational representations of ongoing experience, providing a principled framework for testing how information is represented and transformed across cortical systems. Naturalistic audiovisual narratives are a particularly rich but challenging testbed for these models, requiring integration of multimodal inputs over long temporal horizons and generalization across individuals with substantial response variability. We introduce a multimodal sequence-to-sequence Transformer with a hybrid cross-subject parameterization that predicts cortex-wide parcel-wise fMRI time series autoregressively from visual, audio, language, and vision--language representations. We evaluate the approach on data from the Courtois NeuroMod project, where four deeply-sampled participants viewed six seasons of Friends and four feature-length films during fMRI. Sequence-to-sequence temporal modeling yields consistent improvements over single-frame prediction across cortical networks, with gains extending to novel stimuli. A hybrid architecture that pairs a shared stimulus encoder with lightweight subject-specific decoder components outperforms both fully shared and fully individual models, indicating complementary advantages of learning shared stimulus representations across subjects and fitting individual neural readouts. Finally, we show that in data-scarce settings, hybrid models can be personalized to new individuals with limited fMRI data, demonstrating that multi-subject pretraining serves as a strong inductive prior for building individual-specific encoding models. Together, these results indicate that combining multimodal sequence modeling with a hybrid cross-subject architecture offers a scalable framework for personalized brain encoding under naturalistic conditions.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli".

Jane: The gist This sequence-to-sequence Transformer predicts whole-brain fMRI responses to naturalistic multimodal movies by autoregressively predicting activity from visual, auditory, and language inputs.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now let's talk about the core idea behind this work, "A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli," and who came up with it. The authors are Qianyi He and Yuan Chang Leong from the Data Science Institute at the University of Chicago.

Jane: They’re tackling a problem that moves beyond simple correlation, trying to model how different people react differently to the exact same movie or scene using this sequence modeling framework. It’s about capturing individual brain responses alongside general stimulus features.

Lu: The title itself points to their main contribution: cross-subject prediction of brain responses. They are explicitly trying to account for the fact that while the input stimulus is the same, the resulting neural activity changes depending on who is watching it.

Meng: So, they’re not just building a model that predicts *a* brain response; they are building one where the prediction is tailored to an individual subject, which I think makes their architecture much more sophisticated than what we usually see in these kinds of studies.

Lalam: It's about making the system robust enough so it can learn general patterns from all subjects while still being able to adapt quickly when it meets a new person.

The paper's summary: Tom: So, let’s look at what the authors actually say in their summary. They propose an autoregressive Transformer that predicts fMRI activity by taking these multimodal inputs—visual, auditory, and language—and feeding them into a decoder that builds the brain signal step by step.

Jane: The key is how they feed information into that decoder. They use dual cross-attention mechanisms to pull in both the perceptual details from the current stimulus and some high-level narrative context derived from summaries of the content, like episode descriptions.

Lu: They are using sequences of multimodal context to predict sequences of brain activity, which they say lets them capture long-range temporal structure in both the stimuli and the neural responses simultaneously. That’s a big claim about capturing that deep connection between input and output over time.

Meng: And they’re also doing this by using a shared encoder with subject-specific decoder heads, which is a way to make sure the general parts of the learning are shared across subjects, but the final mapping to brain activity is unique per person.

Lalam: They are using autoregressive decoding with teacher forcing and learnable BOS tokens to predict the entire sequence of fMRI data, which helps stabilize training and lets it generate these long sequences reliably.

The paper's improvements: Tom: When we look at what they suggest as improvements, they focus on three main areas. First is that autoregressive sequence generation for brain activity prediction, which simulates the way subjects actually watch things by predicting the future based on past inputs.

Jane: Second is this multimodal context integration via dual cross-attention, where the decoder attends to both the sensory input and those high-level narrative summaries we talked about earlier. That’s a smart way to give it context beyond just what’s happening right now.

Lu: And third, they tackle inter-subject variability with their hybrid architecture—the shared encoder and subject-specific decoder heads—which is designed to leverage common representations while still allowing for that individual adaptation you mentioned earlier.

Meng: From an engineering standpoint, the sliding window augmentation strategy is also a key part of making this work on real data; they segment stimuli into overlapping temporal chunks of forty time steps, and the output sequence length ends up being thirty-five frames after a hemodynamic delay.

Lalam: Plus, they use teacher forcing with learnable BOS tokens during training, which helps them stabilize the learning process and accelerates convergence when predicting these long sequences.

Conclusion: Tom: So to wrap up on this paper, the idea is that by using this multimodal sequence-to-sequence Transformer model for cross-subject prediction of brain responses to naturalistic stimuli, they can predict whole-brain fMRI activity from visual, auditory, and language inputs.

Jane: They’ve shown it can handle the complexity of these different modalities while keeping track of individual differences between subjects through that hybrid architecture setup. It proves that a shared encoder for general stimulus features combined with subject-specific heads is a viable path forward here.

Lu: The implication is that we can move toward models that capture the temporal dependencies in both complex sensory inputs and the resulting neural activity in a unified, generative way, which opens up new avenues for understanding perception.

Meng: Practically speaking, this shows how you can use these large sequence models to build predictive tools for neuroscience data where individual variability is a big factor. It gives us a framework to predict brain responses tailored to specific people.

Lalam: It means that the advances in these multimodal sequence models can help us create systems that are better at understanding and modeling complex human perception by bridging the gap between what we see or hear and what we actually feel in our brains.

More episodes

← Home