Senior AI Training Systems Architect

Databricks Inc.; San Francisco (CA); On-site; USD 229,000 - 343,000; Full time

Job summary

Databricks, headquartered in San Francisco, is hiring a Senior Staff Software Engineer to lead the AI Runtime training infrastructure. You will shape architecture, drive long-term technical direction, and raise reliability for multi-node GPU training at scale.

You will mentor engineers, collaborate with research, product, and customers, and ensure the developer experience scales for the largest training jobs, while advancing performance and cost-efficiency across GPUs and architectures.

Qualifications

• 15+ years in large-scale distributed systems with GPU training or ML systems.
• Hands-on with training large models using FSDP, DeepSpeed or Megatron; parallelism.
• GPU performance expertise: accelerator architecture, high-speed interconnects, kernels.
• Strong knowledge of training resilience: checkpointing, failure detection, automatic recovery.
• Experience building and running multi-tenant cloud platforms as products.
• Track record setting technical direction with measurable business impact.
• Credible in resolving tough trade-offs; mentoring senior engineers.
• BS in Computer Science or related field (MS/PhD preferred).

Responsibilities

• Drive AI Runtime's architecture and long-term technical direction.
• Define reliability and observability strategy for growing fleet.
• Shape developer experience for large training jobs.
• Anticipate industry trends and choose technical bets.
• Tackle multi-node training: orchestration, GPU scheduling, data loading, checkpointing.
• Improve training speed and cost across architectures and GPUs.
• Collaborate with research, product, and customers to decide what to build next.
• Champion engineering excellence and help hire and grow engineers.

Skills

Distributed systems

GPU training

Mentoring

Multi-tenant cloud platforms

Education

BS in Computer Science or related field

MS/PhD preferred

Tools

FSDP

DeepSpeed

Megatron

Job description

Databricks, headquartered in San Francisco, is hiring a Senior Staff Software Engineer to lead the AI Runtime training infrastructure. You will shape architecture, drive long-term technical direction, and raise reliability for multi-node GPU training at scale.

You will mentor engineers, collaborate with research, product, and customers, and ensure the developer experience scales for the largest training jobs, while advancing performance and cost-efficiency across GPUs and architectures.