Senior ML Infra Research Engineer — Distributed Training

present of AI; San Francisco (CA); On-site; USD 180,000 - 250,000; Full time

Job summary

Chai Discovery in San Francisco is seeking a Senior AI/Data engineer to make large model training fast, stable, and cost efficient. You will work on training code from full models down to GPU kernels, keeping long runs healthy on large GPU clusters.

A strong candidate has distributed training experience, performance profiling, and reliability work such as checkpointing and recovery, plus solid Python and PyTorch or JAX skills.

Qualifications

• Hands-on experience with distributed training across multi-GPU clusters.
• Proficiency in performance profiling and throughput optimization.
• Reliability work including checkpointing and recovery.
• Strong Python and PyTorch or JAX skills.

Responsibilities

• Work on training code from full models down to GPU kernels.
• Make large model training fast, stable and cost efficient.
• Collaborate with researchers to scale runs and test new architectures.
• Ensure long runs on big GPU clusters stay healthy with fault tolerance.

Skills

Distributed training

Performance profiling

Checkpointing and recovery

Python

PyTorch

JAX

Job description

Chai Discovery in San Francisco is seeking a Senior AI/Data engineer to make large model training fast, stable, and cost efficient. You will work on training code from full models down to GPU kernels, keeping long runs healthy on large GPU clusters.

A strong candidate has distributed training experience, performance profiling, and reliability work such as checkpointing and recovery, plus solid Python and PyTorch or JAX skills.