Senior Machine Learning Infrastructure Engineer

True Talent Solutions Partners; United States; Remote; USD 200,000 - 400,000; Full time

Job summary

True Talent Solutions Partners is seeking an engineer to design distributed training and inference infrastructure for AI foundation models, starting with weather. You will build scalable pipelines, support petabyte-scale data, and manage models throughout the ML lifecycle to enable reliable, high-performance compute.

You’ll optimize low-level GPU code, explore parallelization strategies, and apply emerging research to infrastructure.

Qualifications

• 2–10 years of experience building large-scale infrastructure for pretrained foundation models.
• Experience optimizing large-scale training, data, or inference workloads with distributed training frameworks such as FSDP or DeepSpeed.
• Hands-on experience with low-level GPU performance optimization and debugging, including CUDA and JAX.
• Familiarity with cloud platforms (GCP, AWS, or Azure), and with Docker, Kubernetes, Linux, and scalable model serving.
• Background in physical AI or AI for science and a generalist approach across the ML lifecycle.
• Strong problem-solving skills and comfort working in a fast-moving, mission-driven environment.

Responsibilities

• Design, deploy, and maintain distributed ML training and inference clusters.
• Build scalable pipelines for petabyte-scale datasets and manage models throughout the ML lifecycle.
• Research and test training approaches, including parallelization strategies and numerical precision trade-offs.
• Profile and debug low-level GPU operations to improve performance.
• Follow emerging research and apply useful techniques to infrastructure and workflows.

Skills

Distributed training

Large-scale infra

ML lifecycle

Problem-solving

Tools

CUDA

JAX

Docker

Kubernetes

Linux

Job description

About the Role

Join an AI research team developing large foundation models for physical systems, starting with weather. You will build the infrastructure that enables distributed training and reliable, high-performance inference at scale.

What You’ll Do

• Design, deploy, and maintain distributed machine learning training and inference clusters.

• Build scalable pipelines for petabyte-scale datasets and manage models throughout the machine learning lifecycle.

• Research and test training approaches, including parallelization strategies and numerical precision trade-offs.

• Profile and debug low-level GPU operations to improve performance.

• Follow emerging research and apply useful techniques to infrastructure and workflows.

What We’re Looking For

• Two to ten years of experience building large-scale infrastructure for pretrained foundation models, including compute, data, distributed training, and inference.

• Experience optimizing large-scale training, data, or inference workloads, with proficiency in distributed training frameworks such as FSDP or DeepSpeed.

• Hands-on experience with low-level GPU performance optimization and debugging, including CUDA and JAX.

• Familiarity with cloud platforms such as GCP, AWS, or Azure, and with Docker, Kubernetes, Linux, and scalable model serving.

• Background in physical AI or AI for science, such as robotics, biology, or related scientific fields, and a generalist approach across the ML lifecycle.

• Strong problem-solving skills and comfort working in a fast-moving, mission-driven environment.

Compensation & Benefits

Salary range: $200,000 to $400,000 USD annually. Visa sponsorship is not available.

Location

Remote within the United States. Primary location: San Francisco, California.