Senior AI Training Systems Engineer

Databricks Inc.; Mountain View (CA); On-site; USD 160,000 - 225,000; Full time

Job summary

Databricks is seeking a Senior Software Engineer to advance AI Runtime, the world’s leading AI platform. You’ll build core AI training infrastructure, improve reliability and observability, and enhance developer experience for large-scale training workloads.

You will collaborate with researchers, product teams, and customers to shape the next generation of GPU-accelerated training capabilities. You’ll tackle multi-node training challenges, optimize GPU efficiency across architectures, and

Qualifications

• 5+ years in large-scale distributed systems, with experience in GPU training, high-performance computing, or ML systems.
• Experience training large models with FSDP, DeepSpeed, or Megatron, and familiarity with parallelism strategies.
• Working knowledge of GPU performance, including accelerator architecture, high-speed interconnects, and kernels.
• Strong understanding of training resilience patterns, such as checkpointing, failure detection, and automatic recovery.
• A background in building and running multi-tenant cloud platforms as products.
• A track record of owning complex features end to end and seeing them through to customer impact.
• The clarity to explain technical designs in writing and in person, and a reputation as a teammate others learn from.
• BS in Computer Science or a related field (MS or PhD preferred).

Responsibilities

• Build core parts of AI Runtime and contribute to Databricks' technical direction in AI training infrastructure.
• Develop reliability and observability features for a fleet that keeps growing in scale and complexity.
• Improve AI Runtime's developer experience, so the largest training jobs feel as simple as a first experiment.
• Help prepare AI Runtime for new GPU generations and workload types.
• Take on hard multi-node training problems, from orchestration and GPU scheduling to data loading and checkpointing.
• Make training faster and cheaper by raising GPU efficiency across model architectures and hardware generations.
• Work with research, product, and customers on what we build next and own features from design through production.
• Champion engineering excellence, raise the bar, and help the engineers around you grow.]

Skills

Large-scale distributed systems

GPU training

High-performance computing

ML systems

FSDP/DeepSpeed/Megatron

Parallelism strategies

GPU performance

Resilience patterns

Multi-tenant cloud platforms

End-to-end ownership

Education

BS in Computer Science or related field

MS or PhD preferred

Job description

Databricks is seeking a Senior Software Engineer to advance AI Runtime, the world’s leading AI platform. You’ll build core AI training infrastructure, improve reliability and observability, and enhance developer experience for large-scale training workloads.

You will collaborate with researchers, product teams, and customers to shape the next generation of GPU-accelerated training capabilities. You’ll tackle multi-node training challenges, optimize GPU efficiency across architectures, and