Staff AI Training Infra Architect GPU HPC & Multi-Node

The Briefed; Mountain View (CA); On-site; USD 190,000 - 260,000; Full time

Benefits offered by this job

Equity grants

Comprehensive medical, dental, and vis

Generous PTO and flexible work options

Job summary

Databricks is seeking a Staff Software Engineer to work on AI Runtime's core distributed systems. You will tackle GPU reliability, fault-tolerant training, custom kernels, and multi-node orchestration to scale enterprise AI workloads.

You will guide architecture, drive performance optimizations, and mentor engineers across teams while collaborating with product and research to shape the future of serverless GPU training infrastructure.

Qualifications

• 10+ years of professional experience in large-scale distributed systems.
• Bachelor's degree in Computer Science or related quantitative field; Master’s or PhD preferred.
• Experience training large-scale models using distributed parallelism frameworks (FSDP, DeepSpeed, or Megatron).
• Proven ability to lead cross-functional initiatives and mentor engineers.

Responsibilities

• Architecture & Technical Direction: Drive major components of AI Runtime's architecture and shape long-term technical direction.
• Reliability & Observability: Design reliability, failure detection, and observability systems for a multi-tenant GPU cluster fleet.
• Multi-Node Training: Tackle distributed orchestration, GPU scheduling, data loading, and fault-tolerant checkpointing.
• Performance & Efficiency: Optimize training speed and reduce compute cost across architectures, accelerators, and communication layers.
• Developer & Agent Experience: Improve AI Runtime developer experience for engineers and autonomous agents.
• Next-Gen Hardware Strategy: Evaluate hardware bets for new GPU generations and accelerators.
• Cross-Functional Leadership & Mentorship: Partner with research, product, and customers to define roadmaps and mentor seniors.

Skills

Distributed systems

GPU training

High-performance computing

DeepSpeed/Megatron/FSDP

Cloud platforms

Education

Bachelor's degree in Computer Science or related field

Master's or PhD preferred

Tools

NVLink

InfiniBand

CUDA

DeepSpeed

Megatron

FSDP

Job description

Databricks is seeking a Staff Software Engineer to work on AI Runtime's core distributed systems. You will tackle GPU reliability, fault-tolerant training, custom kernels, and multi-node orchestration to scale enterprise AI workloads.

You will guide architecture, drive performance optimizations, and mentor engineers across teams while collaborating with product and research to shape the future of serverless GPU training infrastructure.