ML Infra Engineer: Scale Distributed Training & Research

Doist; San Francisco (CA); On-site; USD 180,000 - 250,000; Full time

Job summary

Doist is seeking an Infrastructure Research Engineer to own the core systems researchers rely on for distributed training, data pipelines, and experiment tooling. You will work directly with researchers, translating scientific needs into robust, scalable systems that operate across thousands of GPUs.

Strong Python and C++ skills, plus hands-on GPU profiling and memory optimization, are essential to accelerate research while maintaining reliability at scale.

Qualifications

• Experience building and operating distributed training systems end-to-end.
• Strong fundamentals in distributed systems, networking, storage, and performance analysis.
• Proficiency in Python and C++ for research-oriented infrastructure.

Responsibilities

• Own the core systems used by researchers for distributed training, experiment orchestration, and data pipelines.
• Scale infrastructure to support thousands of GPUs and large-scale RL training.
• Profile and optimize training throughput across data loading, compute, and memory.
• Build and maintain tools for experiment tracking and reproducibility.
• Diagnose and resolve failures across GPUs, networking, and numerics.

Skills

Distributed training systems

Python

C++

PyTorch

GPU profiling

Memory optimization

Parallelism

Debugging

Tools

PyTorch (systems level)

CUDA

Job description

Doist is seeking an Infrastructure Research Engineer to own the core systems researchers rely on for distributed training, data pipelines, and experiment tooling. You will work directly with researchers, translating scientific needs into robust, scalable systems that operate across thousands of GPUs.

Strong Python and C++ skills, plus hands-on GPU profiling and memory optimization, are essential to accelerate research while maintaining reliability at scale.