Remote ML Training Infrastructure Architect

Dover; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Dover is seeking a Member of Technical Staff for Model Training (infra & platform) to own distributed systems that enable researchers to train models, run large-scale experiments, and deploy AI systems efficiently. This role emphasizes architectural decisions and ownership across GPU infrastructure, inference optimization, and orchestration.

You will drive cross-functional initiatives, design scalable pipelines, and mentor engineers while delivering high-performance training and deployment

Qualifications

• Strong fundamentals in distributed systems, concurrency, networking, and performance engineering.
• Experience designing, building, and operating scalable backend systems or distributed platforms.
• Proficiency in Python and at least one systems language (C++, Rust, or Go).
• Hands-on experience with GPU ML infrastructure, distributed training, or large-scale compute orchestration.

Responsibilities

• Architect, build, and scale shared infrastructure for supervised fine-tuning, RL, and on-policy distillation across language, multimodal, and robotics workloads.
• Design distributed training and execution pipelines coordinating GPU workers, rollout generation, teacher-model inference, reward computation, and simulation environments.
• Develop high-performance infrastructure for model training and inference, optimizing GPU utilization, memory, communication, storage, and distributed execution.
• Build and evolve the core architecture of the agent ecosystem, including runtimes, interfaces, environment integrations, registries, and serving APIs.
• Develop robust systems for experiment tracking, dataset and model versioning, checkpointing, fault recovery, and observability.
• Identify and eliminate infrastructure bottlenecks to reduce turnaround time for complex training workloads.
• Define technical standards and platform interfaces to support collaboration across teams.
• Lead engineering initiatives from design through production, mentoring engineers while owning key components.

Skills

Distributed ML Infra

GPU Performance

Distributed Systems

Technical Leadership

Python

C++

Tools

Python

C++

Rust / Go

Job description

Dover is seeking a Member of Technical Staff for Model Training (infra & platform) to own distributed systems that enable researchers to train models, run large-scale experiments, and deploy AI systems efficiently. This role emphasizes architectural decisions and ownership across GPU infrastructure, inference optimization, and orchestration.

You will drive cross-functional initiatives, design scalable pipelines, and mentor engineers while delivering high-performance training and deployment