Senior ML Infrastructure Engineer — GPU Clusters

Flexion; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Competitive compensation

Relocation sponsorship

Open PTO policy

Job summary

Flexion, a leader in humanoid autonomy, seeks a senior ML engineer to own GPU compute platforms on-site in San Francisco. You will design, bring up, operate, and optimize large GPU clusters, working with AI engineers to boost training speed and hardware utilization.

You will shape short- and long-term GPU compute strategies, collaborate with infrastructure and AI teams, and influence tooling and platform development to accelerate our roadmap.

Qualifications

• Experience with training or inference of large models (billions of parameters) on distributed multi-node GPU hardware.
• Proficiency in Python and working knowledge of PyTorch.
• Deep understanding of distributed training concepts (DDP, FSDP, NCCL).
• Experience with at least one cloud platform (AWS, GCP, Azure) or large-scale on-premises GPU infrastructure.
• Experience with job scheduling and orchestration tools: Slurm and/or Kubernetes/KubeRay.

Responsibilities

• Architect, run and continuously improve existing and future cloud-based GPU clusters. Select the best frameworks and tooling to run our clusters efficiently. Work on cluster provisioning, job schedulers and monitoring systems.
• Help AI engineers optimize their training workloads and maximize hardware utilization using profilers, contributing to our core ML libraries.
• Contribute to short- and long-term GPU compute strategies in collaboration with our AI engineering teams and help execute on them.
• Optimize capacity and cost by exploring multi-cloud strategies and evaluating trade-offs.
• Raise the bar on engineering practices, including testing, code quality, documentation, and system reliability.

Skills

Python

PyTorch

Distributed training

GPU compute

Cloud platforms

Education

Degree in Computer Science/Engineering

Tools

Slurm

Kubernetes/KubeRay

NCCL

Job description

Flexion, a leader in humanoid autonomy, seeks a senior ML engineer to own GPU compute platforms on-site in San Francisco. You will design, bring up, operate, and optimize large GPU clusters, working with AI engineers to boost training speed and hardware utilization.

You will shape short- and long-term GPU compute strategies, collaborate with infrastructure and AI teams, and influence tooling and platform development to accelerate our roadmap.