Distinguished Inference Engineer

Oho Group; San Francisco (CA); On-site; USD 180,000 - 320,000; Full time

Job summary

Oho Group in San Francisco is seeking a senior inference specialist to architect high-performance transformer inference on a new compute platform. You will optimize model execution, graph transformations and runtime behavior across CPUs, GPUs and accelerators.

You will implement or guide performance-critical C++, CUDA or Triton components, improve attention, GEMM and MoE paths, and design KV-cache, batching and memory management to boost latency, throughput and cost per token.

Qualifications

• Hands-on optimization experience for transformer or generative-AI inference.
• Deep experience with low-level implementation in C++, CUDA or Triton.
• Strong profiling and performance-debugging skills.
• Ability to optimize across the full inference stack—from models to kernels to runtimes.

Responsibilities

• Architect high-performance transformer inference on a new compute platform.
• Optimize model execution, graph transformations and runtime behavior.
• Develop or guide performance-critical C++, CUDA or Triton components.
• Improve attention, GEMM, MoE and other critical execution paths.
• Design KV-cache, batching, memory-management and decoding strategies.
• Analyze multi-device and multi-node inference performance.
• Drive improvements in latency, throughput, utilization and cost per token.

Skills

C++

CUDA

Triton

Transformer inference

Profiling

Performance debugging

Tools

CUDA

Triton

Job description

We’re supporting an advanced-compute company building a new hardware and software platform for AI workloads.

They’re looking for an inference specialist who can optimize the complete path from transformer models through execution engines, compilers, runtimes and kernels to multi-accelerator systems.

What you’ll work on

• Architect high-performance transformer inference on a new compute platform
• Optimize model execution, graph transformations and runtime behavior
• Develop or guide performance-critical C++, CUDA or Triton components
• Improve attention, GEMM, MoE and other critical execution paths
• Design KV-cache, batching, memory-management and decoding strategies
• Optimize tensor, pipeline and expert parallelism
• Analyze multi-device and multi-node inference performance
• Drive improvements in latency, throughput, utilization and cost per token

What we’re looking for

• Principal, Distinguished or equivalent senior technical scope
• Direct optimization of transformer or generative-AI inference
• Hands-on low-level implementation in C++, CUDA, Triton or similar
• Deep expertise across multiple connected layers of the inference stack
• Strong profiling and performance-debugging skills
• Experience taking optimizations beyond isolated kernels into complete systems