Principal Inference Engineer

Oho Group; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Oho Group in the San Francisco area is seeking an experienced inference specialist to optimize the full stack—from transformer models through execution engines, compilers, runtimes and kernels—to multi-accelerator systems for AI workloads.

You will architect high-performance transformer inference on a new compute platform, improve latency and throughput, and contribute hands-on low-level work in C++, CUDA or Triton to push the frontier of AI inference.

Qualifications

• Direct optimization of transformer inference required.
• Hands-on low-level implementation in C++, CUDA or Triton.
• Experience optimizing across inference stack layers.

Responsibilities

• Architect high-performance transformer inference on a new compute platform.
• Optimize model execution, graph transformations and runtime behavior.
• Develop or guide performance-critical C++, CUDA or Triton components.
• Improve attention, GEMM, MoE and other critical execution paths.
• Design KV-cache, batching, memory-management and decoding strategies.
• Optimize tensor, pipeline and expert parallelism.
• Analyze multi-device and multi-node inference performance.
• Drive improvements in latency, throughput, utilization and cost per token.

Skills

Transformer inference optimization

Performance profiling

Low-level programming

System-level optimization

Tools

C++

CUDA

Triton

Job description

We’re supporting an advanced-compute company building a new hardware and software platform for AI workloads.

They’re looking for an inference specialist who can optimize the complete path from transformer models through execution engines, compilers, runtimes and kernels to multi-accelerator systems.

What you’ll work on

• Architect high-performance transformer inference on a new compute platform
• Optimize model execution, graph transformations and runtime behavior
• Develop or guide performance-critical C++, CUDA or Triton components
• Improve attention, GEMM, MoE and other critical execution paths
• Design KV-cache, batching, memory-management and decoding strategies
• Optimize tensor, pipeline and expert parallelism
• Analyze multi-device and multi-node inference performance
• Drive improvements in latency, throughput, utilization and cost per token

What we’re looking for

• Principal, Distinguished or equivalent senior technical scope
• Direct optimization of transformer or generative-AI inference
• Hands-on low-level implementation in C++, CUDA, Triton or similar
• Deep expertise across multiple connected layers of the inference stack
• Strong profiling and performance-debugging skills
• Experience taking optimizations beyond isolated kernels into complete systems