ML Engineer - Frontier AI for Scientific Discovery

Biohub; Redwood City (CA); Hybrid; USD 214,000 - 301,000; Full time

Benefits offered by this job

401(k) match

Volunteer time off

Relocation assistance

Job summary

Biohub is seeking an ML Engineer to join a team building frontier AI and biology-focused infrastructure that accelerates scientific discovery. You’ll work with researchers to train and deploy foundational models, optimize data pipelines, and ensure reliable, scalable systems across large GPU clusters.

You’ll contribute to model training, evaluation, deployment, and monitoring, collaborating with researchers to push the boundaries of biology via AI capabilities.

Qualifications

• 5+ years building ML systems or distributed infrastructure at scale.
• Strong Python, PyTorch, and CUDA skills, including custom training loops, with a solid understanding of model architectures, optimization, and evaluation.
• Experience with large scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
• Ability to diagnose performance problems across GPUs, memory, networking, storage, and data loading.
• Experience building large-scale data pipelines and shared research tools; familiarity with scientific formats such as Zarr or HDF5.
• Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
• Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.

Responsibilities

• Collaborate with researchers to train, fine-tune, and evaluate biological foundation models and diagnose training problems.
• Evaluate and curate training and validation datasets at Petabytes scale.
• Design and create libraries and abstractions that researchers and engineers build on.
• Optimize GPU-native data loading for large tensor workloads and resolve hardware utilization bottlenecks across backends.
• Build and maintain reliable distributed pre and post training systems.
• Support production deployment, with tracking of model and dataset artifacts throughout the ML lifecycle.
• Improve model and cluster monitoring, unit and integration testing, and continuous integration.
• Work with partner researchers and engineers to deploy models and supporting infrastructure in external environments.

Skills

ML systems at scale

Python

PyTorch

CUDA

Large-scale distributed systems

Multi-GPU training

Model deployment & CI

Containers & orchestration

Tools

Slurm

Docker

Kubernetes

Job description

Biohub is seeking an ML Engineer to join a team building frontier AI and biology-focused infrastructure that accelerates scientific discovery. You’ll work with researchers to train and deploy foundational models, optimize data pipelines, and ensure reliable, scalable systems across large GPU clusters.

You’ll contribute to model training, evaluation, deployment, and monitoring, collaborating with researchers to push the boundaries of biology via AI capabilities.