Inference Optimization Lead

Up Top; United States; Hybrid; USD 180,000 - 320,000; Full time

Job summary

Up Top seeks a hybrid IC/leader to own the technical strategy for large-scale inference performance and to build and guide a dedicated Inference Optimization team.

You will drive latency reduction, throughput improvements, and cost-per-token optimization across a modern GPU fleet, coordinating with benchmarking and routing across multiple inference engines and emerging hardware platforms.

Qualifications

• 8+ years in performance optimization or HPC
• 5+ years leading engineering teams
• Proficiency in Python, Rust, or Go
• Hands-on experience running production LLM inference engines at high volume
• Depth in modern inference optimization: continuous batching, KV-cache management, speculative decoding, quantization, CUDA graphs, and torch.compile

Responsibilities

• Own the end-to-end technical strategy for inference performance across the platform
• Recruit, build, and lead the Inference Optimization team
• Optimize GPU infrastructure across current and next-gen architectures (e.g. H200, B300)
• Improve latency, throughput, and cost-per-token for production LLM inference workloads
• Build reproducible benchmarking harnesses across inference engines such as vLLM and SGLang
• Tune inference routing and multivariate load-balancing algorithms
• Evaluate emerging optimization techniques — custom CUDA/Triton kernels, attention variants, quantization schemes, and compilation improvements
• Assess emerging inference hardware (FPGAs, ASICs, and custom silicon)

Skills

Performance optimization

Team leadership

Python / Rust / Go

GPU profiling

LLM inference experience

Tools

Nsight Systems

Nsight Compute

PyTorch Profiler

CUDA

torch.compile

Job description

Up Top seeks a hybrid IC/leader to own the technical strategy for large-scale inference performance and to build and guide a dedicated Inference Optimization team.

You will drive latency reduction, throughput improvements, and cost-per-token optimization across a modern GPU fleet, coordinating with benchmarking and routing across multiple inference engines and emerging hardware platforms.