Distributed ML Research Scientist — Frontier-Scale Training

Ifm Us; Sunnyvale (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health benefits

401K Plan

Paid time off

Parental leave

EAP

Job summary

Institute of Foundation Models in Sunnyvale is seeking a senior ML engineer to lead frontier-scale pre-training and systems work. You will set up distributed training across multi-node GPU clusters and implement production-grade kernels in CUDA/Triton.

You will prototype optimizers and attention methods, drive mixed-precision paths, and mentor interns and junior engineers while collaborating with researchers and engineers on end-to-end AI systems.

Qualifications

• 5+ years combined industry or hands-on research experience with large-scale deep-learning training.
• Led at least one large-scale transformer pre-training run.
• Proficient in PyTorch or JAX/Flax and DeepSpeed/FSDP/Megatron-LM.
• Experience with distributed training at scale (100+ GPUs).
• Strong software engineering on large ML codebases; clear design docs.

Responsibilities

• Set up DeepSpeed / FSDP / Megatron-LM across multi-node GPU clusters.
• Create robust launch scripts, checkpoints, and job monitoring (NCCL/GLOO/GPU).
• Prototype optimizers or attention methods in PyTorch/JAX/NumPy and convert to CUDA/Triton kernels.
• Lead mixed-precision training (bf16, fp8, 4-bit) and analyze accuracy vs speed.
• Apply kernel fusion, memory optimization, and performance tuning for throughput.
• Build logging, metrics, and experiment-tracking tools for rapid iteration.
• Design ablation studies and statistical tests; mentor interns and junior engineers.

Skills

5+ years experience

Transformer pre-training

PyTorch

JAX/Flax

DeepSpeed / FSDP / Megatron-LM / MOSA

Tools

Slurm

K8s

Ray

NCCL

CUDA

Triton

Job description

Institute of Foundation Models in Sunnyvale is seeking a senior ML engineer to lead frontier-scale pre-training and systems work. You will set up distributed training across multi-node GPU clusters and implement production-grade kernels in CUDA/Triton.

You will prototype optimizers and attention methods, drive mixed-precision paths, and mentor interns and junior engineers while collaborating with researchers and engineers on end-to-end AI systems.