Senior AI Training Systems Architect

Neon; San Francisco (CA); On-site; USD 229,000 - 343,000; Full time

Job summary

Databricks, the Data and AI company, is seeking a Senior Staff Software Engineer to join the AI Runtime team in San Francisco. You will shape architecture, improve reliability, and drive efficiency for large-scale GPU training infrastructure.

You will mentor engineers and collaborate with research, product, and customers to define what we build next, ensuring scalable, performant systems across multi-node GPU workloads.

Qualifications

• 15+ years in large-scale distributed systems with GPU training or ML systems.
• Hands-on experience training large models with FSDP, DeepSpeed, or Megatron.
• Expertise in GPU performance, including accelerator architecture, high-speed interconnects, and kernels.
• Strong understanding of training resilience patterns, such as checkpointing, failure detection, and automatic recovery.
• Background building and running multi-tenant cloud platforms as products.
• Track record of setting technical direction across teams and delivering business impact.
• Ability to broker technical trade-offs and mentor senior engineers.
• BS in Computer Science or MS/PhD preferred.

Responsibilities

• Drive AI Runtime's architecture and lead Databricks' long-term technical direction in AI training infrastructure.
• Define the reliability and observability strategy for a fleet that keeps growing in scale and complexity.
• Shape AI Runtime's developer experience for people and agents, so even the largest training jobs feel as simple as a first experiment.
• Anticipate industry trends and choose technical bets that prepare AI Runtime for new GPU generations and workload types.
• Take on the hardest multi-node training problems yourself, from orchestration and GPU scheduling to data loading and checkpointing.
• Make training faster and cheaper by raising efficiency across model architectures and GPU accelerators.
• Work with research, product, and customers to decide what we build next and align teams on hard technical calls.
• Champion engineering excellence, raise the bar, and help us hire and grow great engineers.

Skills

Distributed systems

GPU training

FSDP/DeepSpeed

Megatron

GPU performance

Checkpointing

Multi-tenant cloud

Mentorship

CS degree

Education

BS in Computer Science

Job description

Databricks, the Data and AI company, is seeking a Senior Staff Software Engineer to join the AI Runtime team in San Francisco. You will shape architecture, improve reliability, and drive efficiency for large-scale GPU training infrastructure.

You will mentor engineers and collaborate with research, product, and customers to define what we build next, ensuring scalable, performant systems across multi-node GPU workloads.