Director of Engineering, AI Inference Platform

Oho Group; San Francisco (CA); On-site; USD 260,000 - 420,000; Full time

Job summary

Oho Group in San Francisco is seeking a Director of Engineering to build and lead the team responsible for turning its frontier-model inference platform into production-grade software and services.

You will own the architecture and delivery of the end-to-end inference stack, including model enablement, runtimes, scheduling, and serving infrastructure, while mentoring an exceptional team and guiding performance decisions.

Qualifications

• Significant experience building high-performance inference or distributed AI systems.
• Leadership experience across inference, ML systems, accelerator software or performance engineering.
• Strong understanding of transformer inference, attention, KV-cache behaviour, batching and autoregressive decoding.

Responsibilities

• Define the architecture and roadmap for the end-to-end inference stack.
• Build and lead teams across inference, runtimes, kernels and distributed systems.
• Optimise prefill and decode performance across throughput, latency, memory utilisation, power and cost.
• Develop parallelism and scheduling strategies across chips, servers and racks.
• Guide work across continuous batching, KV-cache management, speculative decoding and prefill/decode disaggregation.
• Lead enablement and optimisation of transformer and MoE models.
• Partner with compiler and kernel teams on graph optimisation, operator fusion, tiling and code generation.
• Translate production workloads into improvements across compute, memory and interconnect architecture.
• Establish benchmarking, profiling, observability and performance-regression infrastructure.
• Work directly with frontier AI companies, hyperscalers and cloud customers.

Skills

High-performance inference

Distributed AI systems

Leadership

Transformer inference

Model parallelism

C++

Python

CUDA

Tools

PyTorch

TensorRT-LLM

JAX

XLA

CUDA

Triton

Job description

Oho Group in San Francisco is seeking a Director of Engineering to build and lead the team responsible for turning its frontier-model inference platform into production-grade software and services.

You will own the architecture and delivery of the end-to-end inference stack, including model enablement, runtimes, scheduling, and serving infrastructure, while mentoring an exceptional team and guiding performance decisions.