Learning Transferable Sensor Models via Language-Informed Pretraining
cs.AI, cs.LG
Submitted: 2026-03-12
Updated: 2026-09-27
Code: https://github.com/yuc0805/SLIP
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured from diverse healthcare sensors.
Terminology
Abstract
Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured from diverse healthcare sensors. However, existing sensor-language models can only process a fixed number of sensor channels at a fixed sampling rate, making them difficult to adapt to new or different sensor configurations. This inflexibility prevents models from generalizing across heterogeneous health sensing ecosystems, where data spans high-frequency wearable streams to sparse, day-scale clinical signals. To bridge this gap, we introduce SLIP (Sensor Language-Informed Pretraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing a flexible patch embedder, SLIP transfers to new sensor configurations at inference without retraining, regardless of their native sampling rate or input length. Across 11 datasets, SLIP demonstrates superior performance in retrieval, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 3.56% improvement over the strongest baseline (SensorLM). Beyond classification, SLIP supports open-vocabulary sensor captioning and question answering without any architectural modifications. Our work offers broad implications for the development of sensor-language models, utilizing cross-modal supervision to ensure more robust generalizability in health sensing applications.
Sources
- Large-scale Training of Foundation Models for Wearable Biosignals
- FlexiViT: One Model for All Patch Sizes
- CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition
- SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling
- MOMENT: A Family of Open Time-series Foundation Models
- Masked Autoencoders Are Scalable Vision Learners
- Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
- OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Sundial: A Family of Highly Capable Time Series Foundation Models
- Decoupled Weight Decay Regularization
- Scaling Wearable Foundation Models
- EEG-CLIP : Learning EEG representations from natural language descriptions
- PaPaGei: Open Foundation Models for Optical Physiological Signals
- Learning Transferable Visual Models From Natural Language Supervision
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Gemma 3 Technical Report
- Leveraging Vision-Language Models for Granular Market Change Prediction
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems