Data Infrastructure Engineer for GPU-Scale ML Pipelines

OpenAI; California (MO); On-site; USD 140,000 - 190,000; Full time

Job summary

OpenAI is seeking an engineer to design and implement the dataset infrastructure powering our next-generation training stack. You will build standardized dataset interfaces, scale pipelines across thousands of GPUs, and proactively test performance bottlenecks to keep training at scale.

You will collaborate with multimodal researchers and other infra teams to unify, optimize, and surface datasets for easy consumption, document interfaces for discoverability, and establish safeguards to ensure

Qualifications

• Experience with distributed systems, data pipelines, or infrastructure.
• Strong design sense for scalable APIs and interfaces.
• Ability to diagnose and resolve performance bottlenecks at scale.

Responsibilities

• Design and maintain standardized dataset APIs, including multimodal data that cannot fit in memory.
• Build proactive testing and scale validation pipelines for dataset loading at GPU scale.
• Collaborate with researchers and infra teams to unify and optimize datasets for training and inference.
• Document and maintain dataset interfaces for discoverability and consistent adoption.
• Establish safeguards and validation systems to ensure datasets remain reproducible and unchanged once standardized.
• Debug and resolve performance bottlenecks in distributed dataset loading and surface bottlenecks through tooling.

Skills

Distributed systems

Data pipelines

APIs

Performance debugging

GPU-scale infra

Job description

OpenAI is seeking an engineer to design and implement the dataset infrastructure powering our next-generation training stack. You will build standardized dataset interfaces, scale pipelines across thousands of GPUs, and proactively test performance bottlenecks to keep training at scale.

You will collaborate with multimodal researchers and other infra teams to unify, optimize, and surface datasets for easy consumption, document interfaces for discoverability, and establish safeguards to ensure