Engineering Manager, GPU Inference Performance

Baseten; New York (NY); On-site; USD 240,000 - 270,000; Full time

Benefits offered by this job

Competitive compensation

Meaningful equity

Medical, dental, and vision insurance

Flexible PTO

Parental leave

401(k)

Job summary

Baseten is seeking an Engineering Manager to lead part of our Inference Performance team. This team makes the world's most demanding AI workloads run faster and more efficiently on GPUs.

You'll manage and grow a team of inference performance engineers working across the inference engine and runtime: kernels, scheduling, batching, KV-cache management, speculative decoding and prefill/decode disaggregation. This is a hands-on technical leadership role.

Qualifications

• Bachelor's, Master's or Ph.D. in CS/Engineering/Math or related field.
• Experience hiring, mentoring and conducting performance reviews for engineers.
• Track record leading GPU optimization in training, inference or recommendation systems.
• Strong depth in GPU workloads and architecture tradeoffs.
• Communicates clearly with stakeholders across teams.

Responsibilities

• Lead, mentor and grow a team of inference performance engineers through regular 1:1s and reviews.
• Hire top GPU/inference talent and foster a collaborative culture as the team scales.
• Own the technical roadmap and drive runtime performance work aligning with product goals.
• Review designs, guide profiling/optimizations and reason from first principles about time/memory.
• Advance productionization of techniques like quantization, caching and scheduling.
• Translate performance wins into measurable outcomes (tokens/GPU-hour, latency, cost).
• Collaborate with Infrastructure, Model APIs and customer teams to ship wins.
• Set high standards for quality, benchmarking, and incident response.

Skills

GPU optimization

Team leadership

Hiring & mentoring

PyTorch

TensorRT

Performance profiling

Education

Bachelor's/Master's/Ph.D. in CS/Engineering/Math

Tools

CUDA

Triton

CUTLASS

PyTorch

TensorRT-LLM

Job description

Baseten is seeking an Engineering Manager to lead part of our Inference Performance team. This team makes the world's most demanding AI workloads run faster and more efficiently on GPUs.

You'll manage and grow a team of inference performance engineers working across the inference engine and runtime: kernels, scheduling, batching, KV-cache management, speculative decoding and prefill/decode disaggregation. This is a hands-on technical leadership role.