Senior ML Systems Engineer: Training Frameworks & HPC

Visa Hunt; Greater London; Hybrid; GBP 110,000 - 170,000; Full time

Benefits offered by this job

Weekly lunch stipend

Health and dental benefits

RRSP matching / Pension

Parental Leave top-up

Enrichment benefits (arts, fitness, or

Paid vacation 6 weeks

Travel budget to other offices

Home office stipend

Job summary

Cohere is seeking a senior engineer to help build, maintain and evolve the training framework powering frontier-scale language models. This role sits at the intersection of large-scale training, distributed systems, and HPC infrastructure.

You will design and maintain the core components that enable fast, reliable, and scalable model training — and build the tooling that connects research ideas to thousands of GPUs.

Qualifications

• Experience in large-scale distributed training or HPC systems.
• Familiarity with multi-node cluster orchestration (Slurm, Ray, Kubernetes).
• Ability to debug performance issues across CUDA/NCCL, networking, IO and data pipelines.

Responsibilities

• Build and own the training framework for large-scale LLM training.
• Design distributed training abstractions (data/tensor/pipeline parallelism, memory management, checkpointing).
• Improve training throughput and stability on multi-node clusters.
• Develop tooling for monitoring, logging, debugging, and developer ergonomics.
• Collaborate with infra teams to ensure clusters and hardware configurations support high-performance training.
• Investigate and resolve performance bottlenecks across the ML systems stack.
• Build robust systems for reproducible, debuggable large-scale runs.

Skills

Distributed training

JAX internals

Kubernetes/Ray

CUDA debugging

Containerization

Performance tuning

Collaboration

Tools

Slurm

Ray

Kubernetes

Docker

Apptainer

Job description

Cohere is seeking a senior engineer to help build, maintain and evolve the training framework powering frontier-scale language models. This role sits at the intersection of large-scale training, distributed systems, and HPC infrastructure.

You will design and maintain the core components that enable fast, reliable, and scalable model training — and build the tooling that connects research ideas to thousands of GPUs.