Model Runtime Engineer for Frontier AI Inference

OpenAI; San Francisco (CA); On-site; USD 266,000 - 445,000; Full time

Job summary

OpenAI is seeking a seasoned systems programmer to build the model runtime for frontier models on our AI accelerator. You will design an inference runtime that sits between model execution on hardware and the serving stack, optimizing throughput and latency.

Collaborating across model architecture, distributed systems, compilers and kernel teams, you will push toward production-grade reliability, observability, and scalable performance for OpenAI's supercomputing platform.

Qualifications

• Strong systems programming experience in C++, Rust, or Python.
• Experience building or optimizing runtimes, distributed systems, or compilers.
• Understand LLM inference, including prefill, decode, batching, and model parallelism.
• Ability to reason about latency, throughput, memory bandwidth, and utilization.

Responsibilities

• Design and implement the LLM inference runtime for frontier models on custom silicon.
• Build scheduling, memory management, and execution orchestration for high-performance inference.
• Develop distributed execution strategies across chips, hosts, and racks.
• Optimize end-to-end latency, throughput, and hardware utilization.
• Create profiling, observability, and performance-modeling tools.
• Debug correctness and reliability issues across software and hardware.

Skills

C++

Rust

Python

Distributed systems

Systems programming

Job description

OpenAI is seeking a seasoned systems programmer to build the model runtime for frontier models on our AI accelerator. You will design an inference runtime that sits between model execution on hardware and the serving stack, optimizing throughput and latency.

Collaborating across model architecture, distributed systems, compilers and kernel teams, you will push toward production-grade reliability, observability, and scalable performance for OpenAI's supercomputing platform.