Remote-First Engineering Manager, Inference Performance

Baseten; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Remote-first

Team summits

Unlimited PTO

Healthcare coverage

Parental leave

401(k)

Learning budget

Job summary

Baseten seeks an Engineering Manager to lead the Inference Performance team, driving GPU-focused optimization across the runtime stack. You will manage engineers, unblock hard problems, and set technical direction while growing the team and aligning with broader platform goals.

You will own the roadmap for runtime performance, balance customer needs with new model launches, and coach engineers through strategy, design reviews, and performance tuning.

Qualifications

• Experience managing engineers, including hiring, mentoring and performance reviews.
• Track record of driving roadmaps and shipping complex technical projects with a team.
• Strong technical depth in GPU workloads, with a solid understanding of GPU architecture and performance tradeoffs.
• Clear written and verbal communication, including the ability to align stakeholders across teams.
• Familiarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM.
• Experience leading or closely supporting GPU optimization teams in training, inference or recommendation systems.
• Familiarity with inference engines such as vLLM, SGLang or TensorRT-LLM.
• Experience with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching) in production.
• Experience with GPU kernels (CUDA, Triton, CUTLASS, or similar).
• Experience scaling a team through rapid growth at a startup.
• A background as a hands-on performance or systems engineer before moving into management.

Responsibilities

• Lead, mentor and grow a team of inference performance engineers.
• Own the technical roadmap and execution for runtime performance work.
• Review designs, guide profiling and optimization efforts, and help the team reason from first principles about where time and memory go.
• Drive the productionization of inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill and custom scheduling.
• Turn performance wins into measurable outcomes: tokens per GPU-hour, utilization, latency and cost.
• Partner with Infrastructure, Inference Platform, Kernels, Model APIs and customer-facing teams to set priorities, coordinate launches and ship wins.
• Set high standards for engineering quality, benchmarking, operational excellence and incident response.

Skills

GPU performance

Leadership

Hiring

Mentoring

Performance reviews

Communication

PyTorch familiarity

TensorRT familiarity

CUDA

Team scaling

Education

Bachelor's, Master's, or Ph.D. in CS/Engineering/Math

Tools

vLLM

SGLang

TensorRT-LLM

TensorRT

CUDA

Job description

Baseten seeks an Engineering Manager to lead the Inference Performance team, driving GPU-focused optimization across the runtime stack. You will manage engineers, unblock hard problems, and set technical direction while growing the team and aligning with broader platform goals.

You will own the roadmap for runtime performance, balance customer needs with new model launches, and coach engineers through strategy, design reviews, and performance tuning.