Staff AI Runtime Engineer - Scalable GPU Training Platform

Databricks; California (MO); On-site; USD 190,000 - 265,000; Full time

Job summary

Databricks is seeking a Staff Software Engineer for AI Runtime to scale large‑scale GPU training systems. You will architect the AIR training stack, drive multi-node orchestration, and improve resilience, performance, and developer experience.

You will mentor engineers and align technical work with a long‑term vision across product, research, and platform teams. You will collaborate with diverse teams to enable production training at scale, support accelerator readiness, and expand capabilities

Qualifications

• 10+ years of experience building and operating large-scale distributed systems, with depth in GPU infrastructure or ML systems.
• Hands-on experience with distributed training frameworks and parallelism strategies for large models.
• Strong understanding of training resilience, including checkpointing, failure detection, and automatic recovery.

Responsibilities

• Lead architecture and evolution of AIR's managed GPU training stack to deliver scalable, high-throughput training across fleets.
• Drive multi-node orchestration, distributed parallelism, GPU scheduling, data loading, and long-running job resilience.
• Push GPU efficiency and training performance while reducing cost per training run across architectures and hardware generations.
• Build resilience and observability foundations to keep multi-node jobs healthy and recoverable from failures.

Skills

Distributed training

GPU training infrastructure

PyTorch/DeepSpeed/Megatron

Distributed systems design

Cloud multi-tenant platforms

Performance optimization

Architectural leadership

Mentoring engineers

Strong communication

BS in CS (MS/PhD preferred)

Education

BS in Computer Science or related field

Job description

Databricks is seeking a Staff Software Engineer for AI Runtime to scale large‑scale GPU training systems. You will architect the AIR training stack, drive multi-node orchestration, and improve resilience, performance, and developer experience.

You will mentor engineers and align technical work with a long‑term vision across product, research, and platform teams. You will collaborate with diverse teams to enable production training at scale, support accelerator readiness, and expand capabilities