Senior ML Infra Engineer — GPU Clusters for Humanoid AI

Flexion; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Competitive Compensation

Relocation sponsorship

401(k) with company contributions

Health, dental & vision coverage

Open PTO

Job summary

Flexion seeks a senior ML engineer to own its GPU compute platforms on-site in San Francisco. You will design, bring up, operate and optimize distributed multi-node GPU clusters and collaborate with AI engineers to accelerate training and improve hardware utilization.

You will influence long-term compute strategy with the infrastructure and AI teams, explore multi-cloud options, and raise engineering standards with testing, docs, and reliability practices.

Qualifications

• Hands-on experience with the training or inference of large models (billions of parameters) on distributed multi-node GPU hardware.
• Proficiency in Python and working knowledge of PyTorch.
• Deep understanding of distributed training concepts (DDP, FSDP, NCCL).
• Experience with at least one cloud platform (AWS, GCP, Azure or neoclouds) or large-scale on-premises GPU infrastructure.
• Experience with job scheduling and orchestration tools: Slurm and/or Kubernetes/KubeRay.

Responsibilities

• Architect, run and continuously improve existing and future cloud-based GPU clusters. Select the best frameworks and tooling to run our clusters efficiently. Work on cluster provisioning, job schedulers and monitoring systems
• Help AI engineers optimize their training workloads and maximize hardware utilization using profilers, contributing to our core ML libraries
• Contribute to short- and long-term GPU compute strategies in collaboration with our AI engineering teams and help execute on them.
• Optimize capacity and cost by exploring multi-cloud strategies and evaluating trade-offs
• Raise the bar on engineering practices, including testing, code quality, documentation, and system reliability

Skills

Python

PyTorch

Distributed training

Slurm

Kubernetes

AWS/GCP/Azure

Education

Degree in CS or related

Tools

KubeRay

Profilers

Job description

Flexion seeks a senior ML engineer to own its GPU compute platforms on-site in San Francisco. You will design, bring up, operate and optimize distributed multi-node GPU clusters and collaborate with AI engineers to accelerate training and improve hardware utilization.

You will influence long-term compute strategy with the infrastructure and AI teams, explore multi-cloud options, and raise engineering standards with testing, docs, and reliability practices.