Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules
cs.LG, cs.AI
Submitted: 2026-08-23
Updated: 2026-09-01
Code: https://github.com/Boehringer-Ingelheim/mol-jepa
License: http://creativecommons.org/licenses/by/4.0/
The gist: Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of biochemical
Terminology
Abstract
Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of biochemical environments. To address these challenges, we present Mol-JEPA, a scalable framework for learning molecular world models. Rather than relying on suboptimal molecular perturbations, our model uses modality masking to exploit information from molecular structures, cellular phenotypes, binding affinities, ADMET profiles, quantum chemistry simulations and other drug discovery data. Across various benchmarks, we show that the representations learned by Mol-JEPA deliver strong performance, demonstrating the value of incorporating biochemical context through latent space prediction.
Sources
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- stable-pretraining-v1: Foundation Model Research Made Simple
- Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets
- ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
- Do Larger Models Really Win in Drug Discovery? A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction
- GraphCL: Contrastive Self-Supervised Learning of Graph Representations
- Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development
- Understanding Dimensional Collapse in Contrastive Self-supervised Learning
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning
- Semi-Supervised Classification with Graph Convolutional Networks
- WelQrate: Defining the Gold Standard in Small Molecule Drug Discovery Benchmarking
- MolFM: A Multimodal Molecular Foundation Model
- MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning
- Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
- Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
- Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks