Senior Research Scientist - VLM Pretraining

Epsilon Health; San Francisco (CA); Sur place; USD 180 000 - 260 000; Plein temps

Résumé du poste

Epsilon Health seeks a Research Scientist to lead the pretraining stage of radiology-focused vision-language models. You will own architecture design, data mixtures, and large-scale training across X-ray, CT, and MRI modalities, leveraging a large and diverse medical imaging dataset with clinical rigor.

The role emphasizes scaling, evaluation, and dissemination through publications and internal docs, collaborating with post-training teams to ensure production-quality results for clinical use.

Qualifications

• 6+ years of academia/industry experience in vision-language modeling, multimodal learning, or related fields
• Deep expertise in pretraining large vision-language models and multimodal fusion
• Experience with distributed training and producing production-ready code

Responsabilités

• Design, train, and scale vision-language foundation models for radiology applications, owning the pretraining stage end to end
• Develop VLM architectures suited to medical imaging with high-resolution tiling and token budgets
• Build multimodal pretraining mixtures across captioning, VQA, grounding and retrieval tasks
• Develop fine-grained visual grounding during pretraining and enable localization with boxes or masks
• Own pretraining evaluation and assess base model quality with transfer and fine-tuning analyses
• Train vision-language embeddings with contrastive and generative objectives across image-report pairs
• Contribute to dataset curation, architecture design, distributed training and checkpoint handoffs
• Stay current with vision-language modeling research and publish findings

Connaissances

Vision-language modeling

Multimodal learning

Pretraining large models

Research collaboration

Distributed training

Production-quality code

PyTorch

JAX

Medical imaging knowledge

Outils

PyTorch

JAX

Description du poste

About Us

Epsilon Health is transforming medical imaging and diagnostics by bringing frontier AI into clinical practice. We're tackling some of healthcare's most critical challenges, building technology that directly improves patient outcomes.

This year, we've served over 300,000 patients and now serve 3,000+ daily, working with some of the largest medical institutions in the U.S. By year-end we will interpret 1% of all daily U.S. X-rays.

Our research team brings experience from DeepMind and Meta, powered by one of the industry's most comprehensive proprietary medical imaging datasets, continuously enriched by real-world clinical data. Together, we're building a future where frontier AI helps save millions of lives today, not tomorrow.

Recently out of stealth, we're proudly backed by Jack Altman, Uncork Capital, Renegade Partners, and AlleyCorp, one of NYC's most active healthcare venture firms.

Role Overview

We're seeking a Research Scientist with deep expertise in large-scale vision-language pretraining to join our ML Research team. You'll be at the forefront of developing state-of-the-art multimodal models for clinical use in radiology settings. This role owns the pretraining stage of our radiology report generation model: VLM architecture design, multimodal data and task mixtures, and the large-scale training runs that build grounded visual understanding across X-rays, CT scans, and MRI. You'll work with one of the largest and most diverse medical imaging datasets in the industry, paired with the reports that make multimodal pretraining at this scale possible, while maintaining the clinical rigor required for healthcare deployment. Post-training and RL are owned by a partner role you'll collaborate with closely.

Key Responsibilities

• Design, train, and scale vision-language foundation models for radiology applications, owning the pretraining stage end to end.

• Develop VLM architectures suited to medical imaging, including native and variable resolution handling, high-resolution tiling, connector design, and token budgets for volumetric studies.

• Build and tune multimodal pretraining mixtures across captioning, VQA, grounding, and retrieval tasks, balancing data sources to avoid regressions in language capability.

• Develop fine-grained visual grounding during pretraining, enabling models to localize findings within medical images using bounding boxes or segmentation masks.

• Own pretraining evaluation (zero- and few-shot transfer, probing, and downstream fine-tunability) — as the signal for base model quality.

• Train joint vision-language embedding spaces using contrastive and generative objectives, including region- and sentence-level alignment between images and reports.

• Contribute hands-on to all stages of pretraining including dataset curation, architecture design, distributed training, and handoff of base checkpoints to post-training.

• Stay current with cutting-edge research in vision-language modeling and large-scale multimodal pretraining.

• Drive research and technical excellence through conference publications and technical blog posts, establishing best practices for pretraining medical VLMs at scale.

Qualifications

• 6+ years of academia/industry experience in vision-language modeling, multimodal learning, or related fields

• Deep expertise in pretraining large vision-language models (e.g., LLaVA, Flamingo, CogVLM, Qwen-VL, InternVL, or similar architectures)

• Strong foundation in modern VLM pretraining techniques including:

  • Vision-language connector and fusion architectures (projection, cross-attention, resampler-based)

  • Variable and high-resolution image handling (native resolution, dynamic tiling, token compression)

  • Contrastive and generative objectives for learning joint vision-language embedding spaces

  • Data and task mixture design, including curriculum and mixture-ratio ablations

• Experience with fine-grained visual grounding (referring expression comprehension, phrase grounding, box or mask prediction)

• Track record of implementing complex models from research papers and adapting them to new domains

• Proficiency in PyTorch or JAX, with experience training large models on multi-GPU/distributed systems

• Experience with autoregressive language modeling and long-context training

• Hands-on experience with medical imaging applications, particularly radiology report generation

• Strong software engineering skills and ability to write production-quality code

Preferred Qualifications

• Publications at top-tier conferences (NeurIPS, ICML, ICLR, CVPR, ACL, EMNLP, MICCAI)

• Experience training vision encoders from scratch, or co-designing them with a downstream VLM

• Experience with interleaved image-text pretraining and synthetic recaptioning pipelines

• Experience with 3D medical image processing and temporal modeling

• Familiarity with clinical NLP and medical knowledge representation

• Knowledge of evaluation methodologies for long-form generation, including factuality assessment and hallucination detection

• Experience with model interpretability, explainability, and uncertainty quantification in safety-critical applications