Senior AI Runtime Engineer — Scalable GPU Training

Databricks; California (MO); On-site; USD 160,000 - 225,000; Full time

Job summary

Databricks is hiring a Senior Software Engineer for AI Runtime to architect and scale the GPU training stack. You will drive multi-node orchestration, fault tolerance, and performance optimization for large-scale AI training workloads across GPUs and clusters.

You will mentor engineers, collaborate with product and research teams, and help shape the APIs and tooling that enable customers to launch and monitor production training jobs.

Qualifications

• 5+ years of experience building and operating large-scale distributed systems.
• Experience with distributed training frameworks (PyTorch, FSDP, DeepSpeed, Megatron).
• Strong understanding of training resilience patterns (checkpointing, failure detection, automatic recovery).
• Solid grasp of GPU performance fundamentals (accelerator architecture, NVLink/InfiniBand).
• Experience building and operating managed, multi-tenant cloud platform products.

Responsibilities

• Architect and evolve AIR's managed GPU training platform for scale and resilience.
• Mentor engineers and collaborate across product, research, and platform teams.
• Shape APIs, CLI, and developer experience for launching and monitoring training jobs.
• Lead end-to-end engineering efforts from design to production rollout.
• Contribute to Databricks' AI training infrastructure direction.

Skills

Distributed systems

GPU training infra

Data parallelism

System design

Communication

Mentoring

Education

BS in Computer Science

MS or PhD preferred

Tools

PyTorch

FSDP

DeepSpeed

Megatron

Job description

Databricks is hiring a Senior Software Engineer for AI Runtime to architect and scale the GPU training stack. You will drive multi-node orchestration, fault tolerance, and performance optimization for large-scale AI training workloads across GPUs and clusters.

You will mentor engineers, collaborate with product and research teams, and help shape the APIs and tooling that enable customers to launch and monitor production training jobs.