Senior AI Runtime Engineering Manager: Scale GPU Training

Menlo Ventures

San Francisco (CA)

On-site

USD 228,600 - 297,120

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Databricks is seeking a Senior Engineering Manager to lead the team owning the AIR product experience and its foundational GPU training infrastructure. You will shape customer‑facing capabilities while ensuring scalability, performance, and reliability across multi‑node training at scale.

You will collaborate with platform, product, infrastructure, and research teams to drive end‑to‑end delivery, mentor engineers, and own architectural decisions for state‑of‑the art GPU training solutions with

Qualifications

8+ years of software engineering experience with 3+ years in engineering management.
Track record building and operating managed GPU training infrastructure at scale (100s/1000s GPUs).
Deep familiarity with distributed training frameworks (PyTorch, DeepSpeed, Composer, Megatron-LM) and parallelism strategies (FSDP, tensor/pipeline parallelism).
Experience with training resilience patterns: checkpointing, elastic training, and automated failure recovery for long-running jobs.
Understanding of GPU performance fundamentals including NCCL, interconnect topologies, and memory optimization.
Experience building platform products with clear SLAs where you've owned the customer experience, not just the backend.
Strong cross-functional leadership across platform, product, and research teams, with the ability to lead through ambiguity and deliver complex projects.
Excellent collaboration and communication skills across engineering, product, and research organizations.
BS/MS in Computer Science, Electrical Engineering, or related technical field.

Responsibilities

Lead, mentor, and grow a high‑performing engineering team responsible for the Custom Training product and its foundational infrastructure.
Define and own the product and technical roadmap for AIR, balancing customer experience, functionality, and foundational investments.
Collaborate with product, research, platform, and infrastructure teams, and customers to drive end‑to‑end delivery from ideation to launch.
Drive architectural decisions and product design for managed GPU training at scale.
Advocate for customer needs through direct engagement, translating engineering decisions into product impact.
Build observability and reliability practices for long‑running, multi‑node training jobs, including checkpoint strategies and failure recovery.
Partner with recruiting to attract, hire, and develop top‑tier engineering talent.