AI Runtime Engineering Lead for Scalable GPU Training

United States Digital Space LLC; Mountain View (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

United States Digital Space LLC is seeking a Senior Engineering Manager to lead the team responsible for the AIR product's training infrastructure and customer-facing experience. You will shape the architecture for scalable GPU training, drive the roadmap, and collaborate with product, research, platform, and customers to deliver end-to-end solutions.

You will mentor a high-performing team, implement reliability and observability practices for long-running multi-node jobs, and partner with

Qualifications

• 8+ years of software engineering experience, with 3+ years in engineering management.
• Experience building and operating managed GPU training infrastructure at scale (hundreds to thousands of GPUs).
• Deep familiarity with distributed training frameworks (PyTorch, DeepSpeed, Megatron-LM) and parallelism strategies (FSDP, tensor/pipeline).
• Experience with training resilience: checkpointing, elastic training, failure recovery for long-running jobs.
• Understanding of GPU performance fundamentals (NCCL, interconnects, memory optimization).

Responsibilities

• Lead, mentor, and grow a high-performing engineering team responsible for the Custom Training product and its foundational infrastructure, including distributed training orchestration, cluster lifecycle, fault tolerance, and training efficiency.
• Define and own the product and technical roadmap for AIR, balancing customer experience, functionality, and foundational investments.
• Collaborate with product, research, platform, infrastructure teams, and customers to drive end-to-end delivery, from ideation and prioritization to launch and operation.
• Drive architectural decisions and product design for managed GPU training at scale.
• Advocate for customer needs through direct engagement, ensuring engineering decisions translate to clear product impact.
• Build observability and reliability practices for long-running, multi-node training jobs, including checkpoint strategies, failure recovery, and operational runbooks.
• Partner with recruiting to attract, hire, and develop top-tier engineering talent.

Skills

Leadership

Distributed training

PyTorch

DeepSpeed

Megatron-LM

NCCL

GPU performance

Education

BS/MS in CS/EE

Tools

Cluster orchestration

NVIDIA GPUs

Tensor parallelism

Job description

United States Digital Space LLC is seeking a Senior Engineering Manager to lead the team responsible for the AIR product's training infrastructure and customer-facing experience. You will shape the architecture for scalable GPU training, drive the roadmap, and collaborate with product, research, platform, and customers to deliver end-to-end solutions.

You will mentor a high-performing team, implement reliability and observability practices for long-running multi-node jobs, and partner with