Technical Lead, Distributed Data & Datasets

OpenAI, Inc.; San Francisco (CA); In loco; USD 210.000 - 320.000; Tempo pieno

Descrizione del lavoro

OpenAI is seeking a deeply hands-on Technical Lead to own datasets across the training infrastructure, shaping how training jobs read data, APIs, storage contracts, and versioning. You will align researchers, framework owners, storage teams, and infrastructure partners around a durable data platform.

The role starts as the primary technical owner for dataset reads, operating directly in code and scaling to broader data movement systems across pretraining and multimodal workflows.

Competenze

• Experience with dataset, data loading, storage, or distributed training infra at large scale.
• Hands-on in training infrastructure and data access paths.
• Comfort working across Python and lower-level systems code.

Mansioni

• Design and own a unified dataset read platform for multiple training frameworks.
• Define dataset APIs, storage-format expectations, versioning, and migration paths.
• Build reliability into read path with stateful iteration, caching, and fast restart."
• Develop terminal and web-based visualizers to inspect data late in the pipeline.
• Write and review production code in core data loading, service, caching, and reliability paths.
• Partner with teams across training frameworks, RL, multimodal models, storage, runtime, and cluster infra.

Conoscenze

Distributed systems

Python

Rust

C++

API design

Strumenti

PyTorch

Descrizione del lavoro

OpenAI is seeking a deeply hands-on Technical Lead to own datasets across the training infrastructure, shaping how training jobs read data, APIs, storage contracts, and versioning. You will align researchers, framework owners, storage teams, and infrastructure partners around a durable data platform.

The role starts as the primary technical owner for dataset reads, operating directly in code and scaling to broader data movement systems across pretraining and multimodal workflows.