Head of Inference Optimization & GPU Performance

Venice AI; New York (NY); On-site; USD 270,000 - 330,000; Full time

Job summary

Venice AI in the United States seeks an seasoned engineering leader to own the technical strategy for high-performance inference and to recruit and guide the Inference Optimization Team. You will optimize GPU infrastructure across architectures and drive latency, throughput, and cost per token for large language models at scale.

In this role you will build benchmarking harnesses, collaborate on routing systems, evaluate new techniques and hardware, and mentor a high‑impact team.

Qualifications

• 8+ years in performance optimization or HPC.
• 5+ years leading engineering teams.
• Proficiency in Python, Rust, or Go.
• Bonus: C++/CUDA.
• Hands-on experience with production LLM inference engines (e.g., vLLM, SGLang).

Responsibilities

• Own Venice’s technical strategy for inference performance.
• Recruit and lead the Inference Optimization Team at Venice.
• Optimize Venice's GPU infrastructure across a range of architectures (e.g. H200s, B300s).
• Improve latency, throughput, and cost per token for LLM inference workloads.
• Build reproducible benchmarking harnesses across inference engines (e.g. vLLM, SGLang) to identify the optimal engine, quantization scheme, and parallelism strategy per workload and GPU SKU.
• Work with our inference routing system to optimize multivariate inference load-balancing algorithms.
• Evaluate emerging inference optimization techniques (custom CUDA/Triton kernels), novel attention variants, new quantization schemes, and compilation stack improvements. Hands-on kernel development experience is a strong plus.
• Evaluate emerging inference hardware (FPGAs, ASICs, custom silicon) for viability in Venice's stack.

Skills

Performance optimization

Team leadership

Python

Rust

Go

C++/CUDA

LLM inference engines

Quantization

Distributed inference

GPU profiling

Diffusion models

Tools

vLLM

SGLang

CUDA

Job description

Venice AI in the United States seeks an seasoned engineering leader to own the technical strategy for high-performance inference and to recruit and guide the Inference Optimization Team. You will optimize GPU infrastructure across architectures and drive latency, throughput, and cost per token for large language models at scale.

In this role you will build benchmarking harnesses, collaborate on routing systems, evaluate new techniques and hardware, and mentor a high‑impact team.