Senior AI Runtime Engineer: Scale GPU Training

Fdetimes; Mountain View (CA); On-site; USD 160,000 - 225,000; Full time

Job summary

Databricks is seeking a Senior Software Engineer to build core parts of AI Runtime and scale its AI training infrastructure. You will own features end-to-end, improving reliability, observability, and developer experience for large-scale training jobs.

You will prepare the platform for new GPU generations, tackle complex multi-node training challenges, and work with researchers, product teams, and customers to drive impactful improvements across architectures and accelerators.

Qualifications

• 5+ years in large-scale distributed systems.
• Experience training large models with FSDP, DeepSpeed or Megatron.
• Familiarity with parallelism strategies and GPU performance.
• Knowledge of training resilience patterns: checkpointing, failure detection, automatic recovery.
• Background building and running multi-tenant cloud platforms as products.
• track record owning complex features end-to-end with customer impact.
• Ability to explain technical designs in writing and in person.

Responsibilities

• Build core parts of AI Runtime and contribute to Databricks' technical direction in AI training infrastructure.
• Develop reliability and observability features for a growing fleet.
• Improve AI Runtime's developer experience for large training jobs.
• Prepare AI Runtime for new GPU generations and workload types.
• Tackle multi-node training challenges from orchestration to data loading and checkpointing.
• Increase training speed and cost-efficiency across architectures and accelerators.
• Collaborate with research, product, and customers, owning features from design to production.
• Promote engineering excellence and mentor peers.

Skills

Large-scale distributed systems

GPU training

FSDP/DeepSpeed/Megatron

Parallelism strategies

GPU performance and kernels

Training resilience & checkpointing

Multi-tenant cloud platforms

End-to-end ownership

Written and verbal communication

Education

BS in Computer Science or related field (MS/PhD preferred)

Job description

Databricks is seeking a Senior Software Engineer to build core parts of AI Runtime and scale its AI training infrastructure. You will own features end-to-end, improving reliability, observability, and developer experience for large-scale training jobs.

You will prepare the platform for new GPU generations, tackle complex multi-node training challenges, and work with researchers, product teams, and customers to drive impactful improvements across architectures and accelerators.