Frontier LLM Inference Runtime Engineer

Triwill Group; United States; On-site; USD 180,000 - 240,000; Full time

Job summary

OpenAI is seeking a highly skilled systems programmer to build the model runtime within the inference engine that runs frontier models on OpenAI’s custom silicon. You will bridge between models and the cluster serving software, translating workloads into efficient execution while optimizing throughput, latency, and reliability.

You will collaborate with model architecture, distributed systems, compilers, kernels, and silicon teams to co-design interfaces and remove bottlenecks.

Qualifications

• Experience building or optimizing runtimes and distributed systems.
• Familiarity with model-serving infrastructure and systems software.
• Understanding of LLM inference, batching, KV-cache tradeoffs.

Responsibilities

• Design and implement the LLM inference runtime for frontier models on custom silicon.
• Build scheduling, continuous batching, memory management, KV-cache management, and execution orchestration for high-performance inference.
• Develop distributed execution strategies across chips, hosts, and racks.

Skills

C++

Rust

Python

Distributed systems

Model inference

Job description

OpenAI is seeking a highly skilled systems programmer to build the model runtime within the inference engine that runs frontier models on OpenAI’s custom silicon. You will bridge between models and the cluster serving software, translating workloads into efficient execution while optimizing throughput, latency, and reliability.

You will collaborate with model architecture, distributed systems, compilers, kernels, and silicon teams to co-design interfaces and remove bottlenecks.