ML Research Infra Engineer — GPU, Reproducible Experiments

Axiōma Search; Greater London; Hybrid; GBP 110,000 - 140,000; Full time

Job summary

Stealth AI Lab in London (Hybrid) is seeking a Member of Technical Staff – Infrastructure to own and scale GPU-driven training platforms. You'll build and operate GPU clusters, manage distributed job scheduling, networking, and storage for ML workloads, and improve observability and failure recovery to empower researchers.

Collaborate with researchers to design reliable pipelines for datasets and model weights, and ensure reproducible experiments across large-scale environments.

Qualifications

• Strong Linux and distributed systems fundamentals.
• Experience with Docker, Kubernetes and/or Slurm.
• Strong Python or Go.
• Good understanding of concurrency, networking and failure recovery.
• Familiarity with GPU networking and NCCL.
• Familiarity with Ray, Terraform, Prometheus, Grafana or OpenTelemetry.

Responsibilities

• Build and operate GPU clusters for model training and inference.
• Manage job scheduling, networking and storage for distributed ML workloads.
• Build systems that can run large numbers of training environments concurrently.
• Improve GPU utilisation, startup times and overall platform reliability.
• Build reliable pipelines for datasets, model weights and checkpoints.
• Improve monitoring, debugging and failure recovery.
• Give researchers simple, reproducible ways to run experiments.

Job description

Stealth AI Lab in London (Hybrid) is seeking a Member of Technical Staff – Infrastructure to own and scale GPU-driven training platforms. You'll build and operate GPU clusters, manage distributed job scheduling, networking, and storage for ML workloads, and improve observability and failure recovery to empower researchers.

Collaborate with researchers to design reliable pipelines for datasets and model weights, and ensure reproducible experiments across large-scale environments.