Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra

arXiv:2610.31078 · astro-ph.GA · Submitted 2026-10-22 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Astrophysics Radio. Generated commentary on the latest astrophysics papers.

Vera: We're looking at the paper: "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra".

Jocelyn: Here's the gist of the paper. The gist The study trains a self-supervised multimodal model to recover galaxy stellar masses from sparse JWST data by jointly learning representations from single-band F150W imaging and observed-frame 1–2 µm PRISM spectra, achieving an R2 of ∼ 0.9 at 1 < z ≤ 3 with cross-modal CLIP alignment further improving performance to R2 = 0.90; arXiv:2610.31078v1 Data Construction The researchers constructed a restricted dataset from multiple JWST archives to emulate the sparse band coverage of wide-field surveys, training on single-band F150W imaging and observed-frame 1–2 µm NIRSpec low-resolution spectra The imaging data utilized single-band JWST F150W observations from the OutThere pure-parallel survey (NIRISS) and the NIRCam surveys COSMOS-Web

16: , JADES

17: , and CEERS

18: Cutouts were extracted at 128 × 128 pixels (30 milliarcseconds per pixel, MJy sr−1 units) and centercropped to 100 × 100 after augmentation Spectra were taken from the DAWN JWST Archive (DJA)

19,20: , focusing only on the observed-frame 1–2 µm window (R ∼ 100, a fixed grid of 56 pixels), emulating both a sparse spectral setup and the wavelength coverage of the Roman grism

22: and the Euclid NISP red grisms

23: The matched image-spectrum pairs were grouped by unique DJA source identifier so that multiple cutouts of one source never leak across train/test splits during alignment and downstream tasks Model Architecture The framework employs two modality-specific encoders pre-trained independently in a self-supervised manner on the full imaging and spectroscopic samples The image encoder is a single-channel vision transformer

ViT; 24: pretrained via DINOv2

25: (upper-left panel of Figure 1), which combines the DINO loss, regularized by KoLeo, and the iBOT loss to align the

CLS: token between two networks The spectrum encoder is a transformer autoencoder with a masked-modeling objective

30: , where tokens are tokenized into overlapping flux patches (Appendix B.2), and the loss is weighted by reported uncertainties These modality-specific encoders are then aligned by a CLIP-style head using a contrastive alignment objective to map both modality embeddings into a shared 256-dimensional latent space Performance Evaluation Linear probes were used to benchmark each embedding setup by regressing physical parameters using a single linear transformation on the frozen embeddings Single encoder representations recovered physically meaningful structure, yielding a mean R2 = 0.75 (image encoder) and R2 = 0.87 (spectrum encoder) for catalog redshift over 1 < z ≤ 3 The stellar mass probe on the CLIP-aligned embedding reached R2 = 0.90 (σNMAD = 0.24 dex) at 1 < z ≤ 3, which is a gain of 0.15 over the photometric baseline alone Cross-modal CLIP alignment raised single-modality performance by ∆R2 = +0.038 ± 0.006 for imaging and +0.092 ± 0.01 for spectroscopy Key Findings and Limitations The joint image-spectrum representation achieved R2 = 0.9 in stellar mass, which is a gain of 0.15 over the photometric baseline alone CLIP alignment raised single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference The performance of the stellar mass probe depends strongly on redshift: stable below z ≃ 3.5, strongest at 1.5 3.5 to R2 = 0.61 and R2 = 0.40 in the last two bins The degradation at z > 3.5 occurs because the F150W band and the PRISM window sample the rest-frame ultraviolet, which correlates more with recent star formation rather than with assembled stellar mass Future work will use the full spectral coverage and improved labels to extend this method to higher redshifts Self-supervised models trained on JWST data recover physical information from a sparse setup of single-band F150W imaging and the observed-frame 1–2 µm PRISM window, reaching R2 ∼ 0.9 and σNMAD = 0.24 dex at 1 < z ≤ 3 CLIP alignment further raises single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference Self-supervised representations thus enable physical parameter estimation under sparse wavelength coverage and should extend to broader regimes, from fewer bands to heterogeneous photometric and spectroscopic combinations The data were obtained from the Mikulski Archive for Space Telescopes (MAST) at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS 5-03127 for JWST Software and Disclosure The software used includes astropy

41–43: , Python

44: , NumPy

45: , PyTorch

46: , PyTorch Lightning

47: , grizli

48: , msaexp

21: and Optuna

51: AI tools (GPT-5.6 Sol, GPT-5.5, GPT-5.4, GPT-5.3-codex, OpenAI, and Opus 4., Opus 5, Opus 5.5, Haiku 4.5) were used to assist with scripts for batch retrieval of survey data No AI tools were used to develop the model’s core components or to write any section of the manuscript All AI-assisted suggestions were reviewed, tested where applicable, and adopted by the authors References

1: S.

Vera: L.

Jocelyn: Finkelstein et al.

Vera: “CEERS Key Paper.

Paper summary: Vera: So, we're wrapping up this look at the paper "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra." It’s basically about using a self-supervised model to figure out how to get stellar mass from very sparse JWST data.

Jocelyn: Right, so the main idea is training these two different encoders—one for the image and one for the spectrum—separately, just by letting them learn from massive amounts of data, before they start working together.

Subrahmanyan: And that joint representation then lets them predict physical properties like stellar mass even when you only have a single band of imaging and a low-resolution slice of the spectrum.

Vera: The authors achieved an R squared value around zero point nine zero at redshifts between one and three, which is pretty strong for this kind of limited input. They also showed that adding cross-modal alignment from CLIP really pushes that performance up by zero point one five compared to just using the photometric baseline alone for mass estimation.

Jocelyn: That’s interesting because it shows how combining different types of information—the picture and the spectrum—even in a simple self-supervised way, gives you a real boost in accuracy for things like stellar mass.

Subrahmanyan: From a theoretical side, this suggests that we can get meaningful structure information even when the wavelength coverage is incomplete, which is crucial for understanding how galaxies assemble over cosmic time.

Vera: But they did flag some limits. The performance starts dropping off significantly at redshifts higher than three point five because the F150W band and those one to two micro-meter windows start looking at different parts of the spectrum, which is more about recent star formation rather than the total stellar mass.

Jocelyn: So, what they aren't doing right now is using every single band available, which would definitely help push that accuracy even further for those high-redshift galaxies.

Subrahmanyan: That’s exactly where we need more data and better labels to test if this method can handle the full spectrum or if it really needs that sparse setup.

Conclusion: Vera: So, we're wrapping up this look at "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra." It’s about using a self-supervised model to figure out how to get stellar mass from very sparse JWST data.

Vera: The authors achieved an R squared value around zero point nine zero at redshifts between one and three, which they found to be pretty strong for this kind of limited input. They also showed that adding cross-modal alignment from CLIP really pushes that performance up by zero point one five compared to just using the imaging data on its own.

Jocelyn: That suggests a way forward where we don't necessarily need every single band available to get meaningful physical information about galaxy structure and evolution in the early universe when using AI tools for representation learning.

Vera: But they did flag some limits on their method. The performance starts dropping off significantly at redshifts higher than three and a half because the F150W band and those one to two micro-meter windows start looking at different parts of the spectrum, which is more about recent star formation rather than the total stellar mass.

Jocelyn: So, what they aren't doing right now is using every single band available, which would definitely help push that accuracy even further for those high-redshift galaxies.

Subrahmanyan: That’s exactly where we need more data and better labels to test if this method can handle the full spectrum or if it really needs that sparse setup.

Vera: It’s a really neat demonstration of how AI can extract physical meaning from messy, incomplete observational data. Next up, we're going to look at how this works in practice with their code and see what those results actually mean for the next round of JWST observations.

Yang Cheng, Morgan Fouesneau, Ivelina G. Momcheva, Tobias Buck, Shengxiu Sun

Max-Planck-Institut für Astronomie (MPIA) · Heidelberg University · Interdisciplinary Center for Scientific Computing (IWR) · Department of Astronomy, School of řPhysics, Peking University · Kavli Institute for Astronomy and Astrophysics, Peking University · Institute for Theoretical Physics, Heidelberg University

astro-ph.GA

Submitted: 2026-10-22

Updated: 2026-10-22

Comments: 5 pages, 2 figures. Accepted for the NeurIPS 2026 RPS Workshop. Comments welcome. Detailed descriptions of the data, model training and additional benchmarks are included in the appendix (11 pages, 4 figures)

Code: https://github.com/2003cy/neurips2026_ssl_jwst

Project page: https://dawn-cph.github.io/dja

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The study trains a self-supervised multimodal model to recover galaxy stellar masses from sparse JWST data by jointly learning representations from single-band F150W imaging and

Key concepts

Self-supervised Multimodal Model: This is a machine learning approach where the model learns meaningful features from both images (F150W) and spectra (PRISM) simultaneously without needing pre-labeled data for these specific pairs. It learns how to relate the visual information in one modality to the spectral information in another, creating a unified understanding of galaxy properties.
CLIP Alignment: CLIP is a technique used to align different types of data, like images and text/spectra embeddings. In this model, it aligns the learned representations from the image encoder and spectrum encoder into a shared space. This alignment helps the model better understand how visual features correspond to spectral features for accurate mass estimation.
Stellar Mass Recovery: The primary goal is to accurately predict a galaxy's stellar mass—the total weight of stars in the galaxy. The model uses sparse data (only one band of light and a limited spectral window) to infer this crucial physical property, which is essential for understanding how galaxies grow over cosmic time.
Sparse Data Emulation: The researchers created a training set that mimics real-world JWST limitations by using only single-band F150W imaging and a restricted 1–2 µm spectral window. This tests the model's ability to extract physical information even when the input data is incomplete, which is common in wide-field surveys.

Terminology

Representation Learning — 1× in this paper · explained in 3 episodes

Summary

The gist The study trains a self-supervised multimodal model to recover galaxy stellar masses from sparse JWST data by jointly learning representations from single-band F150W imaging and observed-frame 1–2 µm PRISM spectra, achieving an R2 of ∼ 0.9 at 1 < z ≤ 3 with cross-modal CLIP alignment further improving performance to R2 = 0.90; arXiv:2610.31078v1

Data Construction

The researchers constructed a restricted dataset from multiple JWST archives to emulate the sparse band coverage of wide-field surveys, training on single-band F150W imaging and observed-frame 1–2 µm NIRSpec low-resolution spectra The imaging data utilized single-band JWST F150W observations from the OutThere pure-parallel survey (NIRISS) and the NIRCam surveys COSMOS-Web [16], JADES [17], and CEERS [18] Cutouts were extracted at 128 × 128 pixels (30 milliarcseconds per pixel, MJy sr−1 units) and centercropped to 100 × 100 after augmentation Spectra were taken from the DAWN JWST Archive (DJA) [19,20], focusing only on the observed-frame 1–2 µm window (R ∼ 100, a fixed grid of 56 pixels), emulating both a sparse spectral setup and the wavelength coverage of the Roman grism [22] and the Euclid NISP red grisms [23] The matched image-spectrum pairs were grouped by unique DJA source identifier so that multiple cutouts of one source never leak across train/test splits during alignment and downstream tasks

Model Architecture

The framework employs two modality-specific encoders pre-trained independently in a self-supervised manner on the full imaging and spectroscopic samples The image encoder is a single-channel vision transformer [ViT; 24] pretrained via DINOv2 [25] (upper-left panel of Figure 1), which combines the DINO loss, regularized by KoLeo, and the iBOT loss to align the [CLS] token between two networks The spectrum encoder is a transformer autoencoder with a masked-modeling objective [30], where tokens are tokenized into overlapping flux patches (Appendix B.2), and the loss is weighted by reported uncertainties These modality-specific encoders are then aligned by a CLIP-style head using a contrastive alignment objective to map both modality embeddings into a shared 256-dimensional latent space

Performance Evaluation

Linear probes were used to benchmark each embedding setup by regressing physical parameters using a single linear transformation on the frozen embeddings Single encoder representations recovered physically meaningful structure, yielding a mean R2 = 0.75 (image encoder) and R2 = 0.87 (spectrum encoder) for catalog redshift over 1 < z ≤ 3 The stellar mass probe on the CLIP-aligned embedding reached R2 = 0.90 (σNMAD = 0.24 dex) at 1 < z ≤ 3, which is a gain of 0.15 over the photometric baseline alone Cross-modal CLIP alignment raised single-modality performance by ∆R2 = +0.038 ± 0.006 for imaging and +0.092 ± 0.01 for spectroscopy

Key Findings and Limitations

The joint image-spectrum representation achieved R2 = 0.9 in stellar mass, which is a gain of 0.15 over the photometric baseline alone CLIP alignment raised single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference The performance of the stellar mass probe depends strongly on redshift: stable below z ≃ 3.5, strongest at 1.5 3.5 to R2 = 0.61 and R2 = 0.40 in the last two bins The degradation at z > 3.5 occurs because the F150W band and the PRISM window sample the rest-frame ultraviolet, which correlates more with recent star formation rather than with assembled stellar mass Future work will use the full spectral coverage and improved labels to extend this method to higher redshifts

Conclusion

Self-supervised models trained on JWST data recover physical information from a sparse setup of single-band F150W imaging and the observed-frame 1–2 µm PRISM window, reaching R2 ∼ 0.9 and σNMAD = 0.24 dex at 1 < z ≤ 3 CLIP alignment further raises single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference Self-supervised representations thus enable physical parameter estimation under sparse wavelength coverage and should extend to broader regimes, from fewer bands to heterogeneous photometric and spectroscopic combinations The data were obtained from the Mikulski Archive for Space Telescopes (MAST) at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS 5-03127 for JWST

Software and Disclosure

The software used includes astropy [41–43], Python [44], NumPy [45], PyTorch [46], PyTorch Lightning [47], grizli [48], msaexp [21] and Optuna [51] AI tools (GPT-5.6 Sol, GPT-5.5, GPT-5.4, GPT-5.3-codex, OpenAI, and Opus 4., Opus 5, Opus 5.5, Haiku 4.5) were used to assist with scripts for batch retrieval of survey data No AI tools were used to develop the model’s core components or to write any section of the manuscript All AI-assisted suggestions were reviewed, tested where applicable, and adopted by the authors

References

[1] S. L. Finkelstein et al. “CEERS Key Paper. I. An Early Look into the First 500 Myr of Galaxy Formation with JWST”. In: ApJL 946.1, L13 (Mar. 2023), p. L13 [2] S. Tacchella et al. “JWST NIRCam + NIRSpec: interstellar medium and stellar populations of young galaxies with rising star formation and evolving gas reservoirs”. In: MNRAS 522.4 (July 2023), pp. 6236–6249 [3] J. Matharu et al. “A first look at a complete view of spatially resolved star formation at 1 < z < 1.8 with JWST NGDEEP+FRESCO slitless spectroscopy”. In: A&A 708, A33 (Mar. 2026), A33 [4] R. E. Hviding et al. “The X-Ray Dot: Exotic Dust or a Late-stage Little Red Dot?” In: ApJL 1000.1, L18 (Mar. 2026), p. L18 [5] K. Glazebrook et al. “A massive galaxy that formed its stars at z ≈ 11”. In: Nature 628.8007 (Apr. 2024), pp. 277–281 [6] G. B. Brammer et al. “EAZY: A Fast, Public Photometric Redshift Code”. In: ApJ 686.2 (Oct. 2008), pp. 1503–1513 [7] G. Brammer. eazy-py. Version 0.6.7. Aug. 2023 [8] B. D. Johnson et al., “Stellar Population Inference with Prospector”. In: ApJS 254, 22 (June 2021), p. 22<ref:

Improvements for AI systems

Self supervised training for sparse data allows recovery of galaxy stellar masses from single-band F150W imaging and low resolution spectra, achieving R2 approximately 0.9 at 1 < z ≤ 3 without requiring multi band coverage necessary for traditional SED fitting.
Cross modal CLIP alignment boosts single modality performance by a gain of delta R squared of plus 0.04 for imaging and plus 0.09 for spectroscopy, narrowing the gap to joint modality inference by pulling matched image-spectrum pairs together in a shared latent space with an InfoNCE objective.
The framework enables linear probe recovery of physical parameters like stellar mass from frozen representations, where the joint image-spectrum representation reaches R2 approximately 0.90 in stellar mass, a gain of 0.15 over the photometric baseline alone and ~ 0.1 over F150W photometry with redshift.
The model provides physically meaningful structure from pre trained single encoders, where the linear probe recovers mean R2 values of 0.75 across three Sérsic parameters (n, Re, q) for image encoder representations and 0.87 for catalog redshift over 1 < z ≤ 3 without fine tuning with labeled data.
The system can qualitatively delineate morphology without supervision by checking the attention of the CLS token to patch tokens in the final transformer layer, where individual heads isolate bright cores, outskirts, and background structures in galaxies and spiral sources.

Abstract

A self-supervised multimodal model recovers galaxy stellar masses from sparse JWST data that lack the multi-band coverage required by traditional SED fitting. We train on single-band F150W imaging paired with observed-frame 1-2 μ m PRISM spectra and obtain a joint image-spectrum representation that recovers stellar mass at 1<z 3 with R squared about 0.9 and σ NMAD = 0.24 dex; cross-modal CLIP alignment further raises single-modality performance by ΔR squared = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference. The code and documentation are available at GitHub.