Engineering Manager, GPU Inference Performance

Baseten; San Francisco (CA); On-site; USD 240,000 - 270,000; Full time

Benefits offered by this job

Competitive compensation

Equity

Medical, dental, vision insurance (US)

Flexible PTO including Winter Break

Paid parental leave

Fertility and family-building stipend

401(k) (US)

Learning and networking opportunities

Job summary

Baseten is seeking an Engineering Manager to lead a portion of the Inference Performance team. You will manage a group of inference performance engineers focusing on GPUs, kernels, scheduling, batching, KV-cache management, and related optimizations.

This is a hands-on technical leadership role that shapes team structure, guides deep GPU work, and hires and develops team members. You will collaborate with Infrastructure, Inference Platform, Kernels, Model APIs and customer teams to set

Qualifications

• Bachelor’s, Master’s, or Ph.D. in CS/Engineering or related field.
• Experience managing engineers, including hiring, mentoring, and performance reviews.
• Experience leading GPU optimization teams in training, inference, or recommendation systems.
• Strong technical depth in GPU workloads and GPU architecture.
• Familiarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM.
• Track record of driving roadmaps and shipping complex technical projects with a team.
• Clear written and verbal communication to align stakeholders across teams.

Responsibilities

• Lead, mentor and grow a team of inference performance engineers through regular 1:1s and performance reviews.
• Hire top GPU and inference engineering talent and build a strong team culture as the runtime team scales.
• Own the technical roadmap and execute runtime performance work balancing customer needs and platform investments.
• Review designs, guide profiling and optimization efforts, and reason from first principles about time and memory.
• Drive productionization of inference techniques such as quantization, speculative decoding, and KV-cache reuse.
• Turn performance wins into measurable outcomes: tokens per GPU-hour, utilization, latency and cost.
• Partner with other teams to set priorities, coordinate launches and ship wins.

Skills

People management

Hiring & mentoring

GPU optimization

GPU architecture

PyTorch / TensorRT

Roadmaps & delivery

Communication

Education

Bachelor’s/Master’s/Ph.D. in CS/Engineering

Tools

PyTorch

TensorRT

TensorRT-LLM

Job description

Baseten is seeking an Engineering Manager to lead a portion of the Inference Performance team. You will manage a group of inference performance engineers focusing on GPUs, kernels, scheduling, batching, KV-cache management, and related optimizations.

This is a hands-on technical leadership role that shapes team structure, guides deep GPU work, and hires and develops team members. You will collaborate with Infrastructure, Inference Platform, Kernels, Model APIs and customer teams to set