Remote ML Infrastructure Lead for Large-Scale Training

True Talent Solutions Partners; United States; Remote; USD 200,000 - 400,000; Full time

Job summary

True Talent Solutions Partners is seeking an engineer to design distributed training and inference infrastructure for AI foundation models, starting with weather. You will build scalable pipelines, support petabyte-scale data, and manage models throughout the ML lifecycle to enable reliable, high-performance compute.

You’ll optimize low-level GPU code, explore parallelization strategies, and apply emerging research to infrastructure.

Qualifications

• 2–10 years of experience building large-scale infrastructure for pretrained foundation models.
• Experience optimizing large-scale training, data, or inference workloads with distributed training frameworks such as FSDP or DeepSpeed.
• Hands-on experience with low-level GPU performance optimization and debugging, including CUDA and JAX.
• Familiarity with cloud platforms (GCP, AWS, or Azure), and with Docker, Kubernetes, Linux, and scalable model serving.
• Background in physical AI or AI for science and a generalist approach across the ML lifecycle.
• Strong problem-solving skills and comfort working in a fast-moving, mission-driven environment.

Responsibilities

• Design, deploy, and maintain distributed ML training and inference clusters.
• Build scalable pipelines for petabyte-scale datasets and manage models throughout the ML lifecycle.
• Research and test training approaches, including parallelization strategies and numerical precision trade-offs.
• Profile and debug low-level GPU operations to improve performance.
• Follow emerging research and apply useful techniques to infrastructure and workflows.

Skills

Distributed training

Large-scale infra

ML lifecycle

Problem-solving

Tools

CUDA

JAX

Docker

Kubernetes

Linux

Job description

True Talent Solutions Partners is seeking an engineer to design distributed training and inference infrastructure for AI foundation models, starting with weather. You will build scalable pipelines, support petabyte-scale data, and manage models throughout the ML lifecycle to enable reliable, high-performance compute.

You’ll optimize low-level GPU code, explore parallelization strategies, and apply emerging research to infrastructure.