Engineering Manager, AI Inference Platform & ML Infra

Google; Sunnyvale (CA); On-site; USD 207,000 - 301,000; Full time

Job summary

Google Sunnyvale is seeking an Engineering Manager for the DSC AI Inference Platform to lead a team of systems and ML engineers. You will shape technical goals, manage delivery, and mentor engineers across multiple sites and disciplines.

You will drive architecture for disaggregated serving, optimize LLM performance on GPU accelerators, and partner with Research, SRE, and Core GPU teams to deploy AI models at scale.

Qualifications

• Bachelor's degree or equivalent practical experience.
• 8 years of experience programming in C++ or Python.
• 5 years of experience optimizing, profiling, and scaling production‑grade systems on GPU accelerators or specialized AI hardware.
• 5 years of experience directly managing and leading engineering teams focused on machine learning infrastructure, AI platforms, or high‑performance distributed computing systems.
• 5 years of experience in a people management or team leadership role.
• 3 years of experience managing engineering organizations across multi‑team infrastructure dependencies.

Responsibilities

• Lead, mentor, and grow a high‑performing team of systems and ML engineers across multiple sites.
• Define the technical goal and strategy for enhancing the LLM serving stack, focusing on performance, scalability, and resource efficiency.
• Drive the design and implementation of advanced serving architectures, including disaggregated serving, to optimize resource utilization and latency.
• Oversee the building and maintenance of critical infrastructure and tooling for in‑depth performance analysis, profiling, and benchmarking of LLM models on GPU accelerators.
• Partner with Research, SRE, Product, and Core GPU library teams to optimize and deploy LLMs in production globally.

Skills

C++

Python

GPU optimization

ML infrastructure

People management

Multi-team leadership

Engineering leadership

Education

Bachelor's degree or equivalent practical experience

Master’s or PhD preferred

Tools

Nsight

xprof

JAX

PyTorch

TensorFlow

CUDA

Job description

Google Sunnyvale is seeking an Engineering Manager for the DSC AI Inference Platform to lead a team of systems and ML engineers. You will shape technical goals, manage delivery, and mentor engineers across multiple sites and disciplines.

You will drive architecture for disaggregated serving, optimize LLM performance on GPU accelerators, and partner with Research, SRE, and Core GPU teams to deploy AI models at scale.