Senior Transformer Inference Architect

Oho Group; San Francisco (CA); On-site; USD 180,000 - 320,000; Full time

Job summary

Oho Group in San Francisco is seeking a senior inference specialist to architect high-performance transformer inference on a new compute platform. You will optimize model execution, graph transformations and runtime behavior across CPUs, GPUs and accelerators.

You will implement or guide performance-critical C++, CUDA or Triton components, improve attention, GEMM and MoE paths, and design KV-cache, batching and memory management to boost latency, throughput and cost per token.

Qualifications

• Hands-on optimization experience for transformer or generative-AI inference.
• Deep experience with low-level implementation in C++, CUDA or Triton.
• Strong profiling and performance-debugging skills.
• Ability to optimize across the full inference stack—from models to kernels to runtimes.

Responsibilities

• Architect high-performance transformer inference on a new compute platform.
• Optimize model execution, graph transformations and runtime behavior.
• Develop or guide performance-critical C++, CUDA or Triton components.
• Improve attention, GEMM, MoE and other critical execution paths.
• Design KV-cache, batching, memory-management and decoding strategies.
• Analyze multi-device and multi-node inference performance.
• Drive improvements in latency, throughput, utilization and cost per token.

Skills

C++

CUDA

Triton

Transformer inference

Profiling

Performance debugging

Tools

CUDA

Triton

Job description

Oho Group in San Francisco is seeking a senior inference specialist to architect high-performance transformer inference on a new compute platform. You will optimize model execution, graph transformations and runtime behavior across CPUs, GPUs and accelerators.

You will implement or guide performance-critical C++, CUDA or Triton components, improve attention, GEMM and MoE paths, and design KV-cache, batching and memory management to boost latency, throughput and cost per token.