Principal Transformer Inference Engineer - High Performance

Oho Group; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Oho Group in the San Francisco area is seeking an experienced inference specialist to optimize the full stack—from transformer models through execution engines, compilers, runtimes and kernels—to multi-accelerator systems for AI workloads.

You will architect high-performance transformer inference on a new compute platform, improve latency and throughput, and contribute hands-on low-level work in C++, CUDA or Triton to push the frontier of AI inference.

Qualifications

• Direct optimization of transformer inference required.
• Hands-on low-level implementation in C++, CUDA or Triton.
• Experience optimizing across inference stack layers.

Responsibilities

• Architect high-performance transformer inference on a new compute platform.
• Optimize model execution, graph transformations and runtime behavior.
• Develop or guide performance-critical C++, CUDA or Triton components.
• Improve attention, GEMM, MoE and other critical execution paths.
• Design KV-cache, batching, memory-management and decoding strategies.
• Optimize tensor, pipeline and expert parallelism.
• Analyze multi-device and multi-node inference performance.
• Drive improvements in latency, throughput, utilization and cost per token.

Skills

Transformer inference optimization

Performance profiling

Low-level programming

System-level optimization

Tools

C++

CUDA

Triton

Job description

Oho Group in the San Francisco area is seeking an experienced inference specialist to optimize the full stack—from transformer models through execution engines, compilers, runtimes and kernels—to multi-accelerator systems for AI workloads.

You will architect high-performance transformer inference on a new compute platform, improve latency and throughput, and contribute hands-on low-level work in C++, CUDA or Triton to push the frontier of AI inference.