Distributed ML Training Performance Engineer

OpenAI; California (MO); Hybrid; USD 170,000 - 260,000; Full time

Benefits offered by this job

Relocation assistance

Job summary

OpenAI is seeking a Training Performance Engineer in San Francisco, CA, to push throughput and uptime across our distributed training stack. You will analyze large-scale runs, pinpoint bottlenecks, and design practical optimizations that scale with growing model size while maintaining compute efficiency.

You will work closely with runtime and systems engineers, researchers, and product teams to implement kernel optimizations, scheduling improvements, and data movement strategies.

Qualifications

• Experience analyzing and optimizing distributed training workloads.
• Strong programming skills in Python and C++.
• Familiarity with PyTorch, JAX, or TensorFlow.
• Experience running multi-GPU systems or HPC clusters.
• Ability to translate profiling data into engineering improvements.

Responsibilities

• Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
• Optimize GPU utilization and throughput for large-scale distributed model training.
• Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
• Implement model graph transforms to improve end to end throughput.
• Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
• Partner with researchers to ensure new model architectures scale efficiently during pre-training.
• Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.

Skills

Python

C++

Rust

CUDA

Tools

NCCL

MPI

UCX

PyTorch

JAX

TensorFlow

Job description

OpenAI is seeking a Training Performance Engineer in San Francisco, CA, to push throughput and uptime across our distributed training stack. You will analyze large-scale runs, pinpoint bottlenecks, and design practical optimizations that scale with growing model size while maintaining compute efficiency.

You will work closely with runtime and systems engineers, researchers, and product teams to implement kernel optimizations, scheduling improvements, and data movement strategies.