Data Infrastructure Engineer - Scalable ML Training Platform

OpenAI; San Francisco (CA); On-site; USD 250,000 - 380,000; Full time

Job summary

OpenAI in San Francisco is seeking an engineer to design and implement dataset infrastructure for next-generation training stacks. You will develop standardized dataset interfaces and scale pipelines across thousands of GPUs.

The role involves close collaboration with multimodal researchers to ensure efficiency and user experience. Ideal candidates have strong engineering fundamentals and experience in distributed systems.

Compensation ranges from $250K to $380K annually.

Qualifications

• Strong engineering fundamentals with experience in distributed systems, data pipelines, or infrastructure.
• Experience building APIs, modular code, and scalable abstractions.
• Comfortable debugging bottlenecks across large fleets of machines.

Responsibilities

• Design and maintain standardized dataset APIs.
• Build testing and scale validation pipelines for dataset loading.
• Collaborate with teams to integrate datasets into training pipelines.
• Document and maintain dataset interfaces for discoverability.
• Establish validation systems to ensure dataset reproducibility.
• Debug performance bottlenecks in distributed dataset loading.
• Provide tools to visualize and inspect datasets.

Skills

Distributed systems

Data pipelines

API building

Debugging skills

Job description

OpenAI in San Francisco is seeking an engineer to design and implement dataset infrastructure for next-generation training stacks. You will develop standardized dataset interfaces and scale pipelines across thousands of GPUs.

The role involves close collaboration with multimodal researchers to ensure efficiency and user experience. Ideal candidates have strong engineering fundamentals and experience in distributed systems.

Compensation ranges from $250K to $380K annually.