Staff Inference Engineer — GPU-Optimized ML Serving

Uniphore; Palo Alto (CA); On-site; USD 190,000 - 240,000; Full time

Benefits offered by this job

Annual incentive

Pre-IPO stock options

Medical, dental, vision

401(k) with company match

Generous paid time off

Job summary

Uniphore, a leading Business AI company, seeks a Staff Inference Engineer to own the inference layer across cloud and on-premises. You will implement context-aware runtimes, optimize KV-cache and prompt-prefix handling, and drive high-performance multi-GPU deployments.

This hands-on role collaborates with research, model engineering, and platform teams. You will contribute to autoregressive inference, model lifecycle management, and scalable serving with zero-downtime upgrades.

Qualifications

• 5+ years engineering with GPU systems or ML serving.
• Experience with vLLM, SGLang, or TensorRT-LLM preferred.
• Strong Python and C++ (C++17/20); CUDA/C++ debugging.

Responsibilities

• Context-aware runtime: integrate context selection and model execution.
• KV cache management, routing, eviction, and offloading.
• Optimize inference: speculative decoding, batching, and quantization.
• Lifecycle: manage models/adapters with zero-downtime swaps.
• Observability and reliability: DCGM, Prometheus, Grafana integration.

Skills

Python

C++ (C++17/20)

GPU systems

CUDA / HIP kernels

Model inference

Profiling & optimization

vLLM / SGLang / TensorRT-LLM

PyTorch internals

Multi-GPU scaling

Education

Bachelor's or higher in CS/EE

Tools

CUDA

HIP

Triton

CUTLASS

Job description

Uniphore, a leading Business AI company, seeks a Staff Inference Engineer to own the inference layer across cloud and on-premises. You will implement context-aware runtimes, optimize KV-cache and prompt-prefix handling, and drive high-performance multi-GPU deployments.

This hands-on role collaborates with research, model engineering, and platform teams. You will contribute to autoregressive inference, model lifecycle management, and scalable serving with zero-downtime upgrades.