Engineering Manager (Inference Performance)

Baseten; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Remote-first

Team summits

Unlimited PTO

Healthcare coverage

Parental leave

401(k)

Learning budget

Job summary

Baseten seeks an Engineering Manager to lead the Inference Performance team, driving GPU-focused optimization across the runtime stack. You will manage engineers, unblock hard problems, and set technical direction while growing the team and aligning with broader platform goals.

You will own the roadmap for runtime performance, balance customer needs with new model launches, and coach engineers through strategy, design reviews, and performance tuning.

Qualifications

• Experience managing engineers, including hiring, mentoring and performance reviews.
• Track record of driving roadmaps and shipping complex technical projects with a team.
• Strong technical depth in GPU workloads, with a solid understanding of GPU architecture and performance tradeoffs.
• Clear written and verbal communication, including the ability to align stakeholders across teams.
• Familiarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM.
• Experience leading or closely supporting GPU optimization teams in training, inference or recommendation systems.
• Familiarity with inference engines such as vLLM, SGLang or TensorRT-LLM.
• Experience with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching) in production.
• Experience with GPU kernels (CUDA, Triton, CUTLASS, or similar).
• Experience scaling a team through rapid growth at a startup.
• A background as a hands-on performance or systems engineer before moving into management.

Responsibilities

• Lead, mentor and grow a team of inference performance engineers.
• Own the technical roadmap and execution for runtime performance work.
• Review designs, guide profiling and optimization efforts, and help the team reason from first principles about where time and memory go.
• Drive the productionization of inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill and custom scheduling.
• Turn performance wins into measurable outcomes: tokens per GPU-hour, utilization, latency and cost.
• Partner with Infrastructure, Inference Platform, Kernels, Model APIs and customer-facing teams to set priorities, coordinate launches and ship wins.
• Set high standards for engineering quality, benchmarking, operational excellence and incident response.

Skills

GPU performance

Leadership

Hiring

Mentoring

Performance reviews

Communication

PyTorch familiarity

TensorRT familiarity

CUDA

Team scaling

Education

Bachelor's, Master's, or Ph.D. in CS/Engineering/Math

Tools

vLLM

SGLang

TensorRT-LLM

TensorRT

CUDA

Job description

• We're looking for an Engineering Manager to lead part of our Inference Performance team. This team makes the world's most demanding AI workloads run faster and more efficiently on GPUs
• You'll manage and grow a team of inference performance engineers working across the inference engine and runtime: kernels, scheduling, batching, KV-cache management, speculative decoding and prefill/decode disaggregation
• This is a hands-on technical leadership role. You'll set direction, unblock hard problems and earn the team's trust by going deep on GPU performance, while also hiring, developing and supporting the people doing the work
• Your team's output directly affects how fast our customers' models run and how efficiently we serve them. The team is scaling quickly, so you'll help shape how it is structured as it grows
• Lead, mentor and grow a team of inference performance engineers through regular 1:1s, clear feedback, career development and performance reviews
• Hire top GPU and inference engineering talent, and build a strong, collaborative team culture as the runtime team scales
• Own the technical roadmap and execution for runtime performance work, balancing customer needs, new model launches and long-term platform investments
• Stay close to the technical work. Review designs, guide profiling and optimization efforts, and help the team reason from first principles about where time and memory go
• Drive the productionization of inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill and custom scheduling
• Turn performance wins into measurable outcomes: tokens per GPU-hour, utilization, latency and cost
• Help the team bring up and tune new model architectures on new hardware quickly, often in the same week they're released
• Partner with Infrastructure, Inference Platform, Kernels, Model APIs and customer-facing teams to set priorities, coordinate launches and ship wins
• Set high standards for engineering quality, benchmarking, operational excellence and incident response

Benefits

• Remote-first work environment. The Baseten team is welcome to work from wherever they want; fully remote, in our San Francisco office, or a mix of both. Today, our team (including our founding team) is spread across the United States, Canada, and Armenia. We provide a $1,000 stipend for you to make your home-office comfortable and productive
• Regular in-person team summits. We get together as a team three times a year to plan, workshop, and most importantly, get to know each other better
• Unlimited PTO. We ask that everyone take at least 4 weeks of vacation. And we have a company-wide break between Christmas and New Year's Day
• Full healthcare coverage. Medical, dental and vision insurance for you and your family
• Paid parental leave. 16-weeks fully paid parental leave (adoptive and non-birth parents included) and flexibility with schedules while returning to work
• Company-sponsored 401(k) for you to contribute to
• Learning and development budget. We encourage you to take classes, attend conferences, and invest in your craft and we'll cover expenses to make it happen

Experience managing engineers, including hiring, mentoring, giving feedback and running performance reviewsA track record of driving roadmaps and shipping complex technical projects with a teamStrong technical depth in GPU workloads, with a solid understanding of GPU architecture and performance tradeoffsClear written and verbal communication, including the ability to align stakeholders across teamsFamiliarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or a related fieldExperience leading or closely supporting GPU optimization teams in training, inference or recommendation systemsFamiliarity with inference engines such as vLLM, SGLang or TensorRT-LLMExperience with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching) in productionExperience with GPU kernels (CUDA, Triton, CUTLASS, or similar)Experience scaling a team through rapid growth at a startupA background as a hands-on performance or systems engineer before moving into management