Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

summary

Video file (mp4)

The gist

The gist: Self-supervised pre-training, combining masked autoencoder reconstruction with relational voxel-level objectives, enables a sparse Vision Transformer framework to learn reusable

In short

The research developed a sparse Vision Transformer framework to learn reusable representations from complex, heterogeneous neutrino detector data. By combining masked autoencoder reconstruction with relational voxel-level objectives, the model learns meaningful features for energy-frontier events, improving performance and data efficiency across multiple downstream physics tasks.

Key concepts

Sparse Vision Transformer (ViT)
A specific neural network architecture designed to process complex, high-dimensional data efficiently. This framework uses a sparse approach combined with self-attention mechanisms to handle the diverse inputs from different detector components like calorimeters and trackers without requiring dense connections across all data points.
Masked Autoencoder (MAE)
A self-supervised pre-training technique where the model learns to reconstruct missing parts of an input. In this work, it is used to train the encoder by masking patches of detector data and forcing the model to predict what those missing parts should look like, establishing a strong initial representation.
Relational Voxel-level Objectives
Additional training targets that go beyond simple reconstruction. These objectives teach the model relationships between different parts of the detector, such as predicting voxel labels or particle categories based on surrounding information. This helps the model understand the physical structure and context of neutrino interactions.
Heterogeneous Detector Data Fusion
The process of combining data from different detector types—like 3DCal, AHCAL, ECAL, and muon spectrometer—into a single representation. The framework uses techniques like Perceiver-IO fusion to integrate these distinct data streams into a unified set of tokens for the Transformer encoder.

Terminology used across episodes

This episode discusses

The paper

Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training · Read on arXiv

IPA, ETH Zurich

Accelerator-based neutrino physics is entering an energy-frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures. In this regime, event interpretation becomes impractical for conventional reconstruction approaches, particularly when labelled data are scarce and the analysis spans diverse downstream objectives. We present a sparse ViT framework for learning reusable representations from heterogeneous detector data. Self-supervised pre-training combines masked autoencoder reconstruction with relational voxel-level objectives for hierarchy, ghost and particle identification, and the resulting shared encoder is then jointly fine-tuned across classification and regression tasks. Evaluated on simulated events from the proposed FASERCal concept at the LHC, we find that pre-training consistently improves neutrino flavour and charm-quark identification, momentum regression, and vertex reconstruction over training from scratch, with the addition of relational objectives yielding further gains in the most topologically complex channels. Interpretability analyses further show that pre-training yields a more structured latent space, while detector-subsystem ablations recover physically plausible channel-dependent roles for the heterogeneous inputs. A data-efficiency study shows that, with roughly 10 cubed labelled events, the pre-trained encoder already matches the flavour-classification performance of a randomly initialised model trained on an order of magnitude more data. The learned representations also transfer effectively to publicly available benchmarks spanning different detector technologies and energy scales, matching or exceeding published baselines. These results support self-supervised pre-training on multimodal detector data as a scalable route towards reusable representations for neutrino and particle-detector analysis.

DOI: 10.1038/s42256-026-01309-6

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training".

Jane: The gist: Self-supervised pre-training, combining masked autoencoder reconstruction with relational voxel-level objectives, enables a sparse Vision Transformer framework to learn reusable representations from heterogeneous detector data for energy-frontier neutrino detectors.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Alright, so what's the main thrust of "Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training"? The paper argues that traditional event interpretation is impractical at the TeV scale because the signatures are just too dense and overlapping.

Jane: They propose a sparse Vision Transformer framework to solve this by learning reusable representations directly from heterogeneous detector data, which is tough when labeled data is scarce or analyses need to cover many different physics goals.

Lu: The thesis involves a two-phase pre-training strategy: first, masked autoencoder reconstruction alone, and then adding relational voxel-level targets for things like ghost identification and particle categorization.

Meng: They introduce a sparse encoder design that mixes sparse convolutional patch embeddings with module-aware self-attention and Perceiver-IO fusion to handle the different detector streams efficiently.

Lalam: The pre-training uses masked reconstruction to map decoder tokens back to voxels, and then adds a relational forward pass where the encoder predicts hierarchy labels, ghost labels, and particle categories on kept three deeCal patches <ref:2604.07037#pg2>.

Tom: So why does this matter for the physics side? They claim that this composite objective improves downstream performance beyond just using masked reconstruction alone, with the largest gains in the most challenging channels.

Jane: And they show that this learned representation is useful across a suite of tasks, including flavour identification, charm-quark identification, and even event kinematics and vertex reconstruction after fine-tuning.

Lu: What's interesting is that they demonstrate this representation transfers beyond the source domain to public benchmarks covering different detector technologies and energy regimes.

Meng: That transfer capability suggests the learned structure isn't just tailored to one specific experiment but captures something more general about dense interaction topologies.

Conclusion: Tom: So, looking at "Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training," the title itself points to building foundational models specifically for these complex, high-energy neutrino detectors using self-supervised training.

Jane: The authors are Sa´ul Alonso-Monsalve, Fabio Cufino, Umut Kose, Anna Mascellani, and Andr´e Rubbia from ETH Zurich. They focus on creating a sparse Vision Transformer framework that learns reusable representations from heterogeneous detector data.

Lu: What this means simply is that they've laid out a path toward general detector encoders for neutrino physics by using masked autoencoder reconstruction alongside relational voxel-level objectives to learn structure at the fundamental level.

Meng: For practical applications, it suggests we can use these pre-trained encoders to start with initial classifications or reconstructions on new detector data much faster than training from scratch.

Lalam: It sets up a concrete path toward more general detector encoders and motivates further work on hybrid pre-training objectives and domain adaptation for these complex environments.

More episodes

← Home