Senior ML Training Frameworks & Tools Engineer

Cohere; New York (NY); Remote; USD 180,000 - 280,000; Full time

Benefits offered by this job

Health & dental benefits

Parental leave top-up

6 weeks vacation

Job summary

Cohere is seeking a senior engineer to build, maintain and evolve the training framework that powers frontier-scale language models. You will design core components for fast, reliable, and scalable model training and create tooling that connects research ideas to thousands of GPUs.

The role spans the full ML systems stack, with opportunities to impact data loading, performance profiling, monitoring, reproducibility, and fault-tolerant checkpointing.

Qualifications

• Strong engineering experience in large-scale distributed training or HPC systems.
• Deep familiarity with JAX internals, distributed training libraries, or custom kernels/fused ops.
• Experience with multi-node cluster orchestration (Slurm, Ray, Kubernetes, or similar).
• Comfort debugging performance issues across CUDA/NCCL, networking, IO, and data pipelines.
• Experience with containerized environments (Docker, Singularity/Apptainer).
• A track record of building tools that increase developer velocity for ML teams.

Responsibilities

• Build and own the training framework for large-scale LLM training.
• Design distributed training abstractions (data/tensor/pipeline parallelism, FSDP/ZeRO, memory management, checkpointing).
• Improve training throughput and stability on multi-node clusters.
• Develop tooling for monitoring, logging, debugging, and developer ergonomics.
• Collaborate with infra to support high-performance training environments.
• Investigate and resolve performance bottlenecks across the ML stack.
• Build robust, reproducible, debuggable large-scale runs.

Skills

Distributed training

JAX internals

CUDA/NCCL debugging

Containerized environments

Developer velocity tooling

Collaboration

Trade-off judgment

Tools

Slurm

Ray

Kubernetes

Docker

Apptainer/Singularity

NCCL

Job description

Cohere is seeking a senior engineer to build, maintain and evolve the training framework that powers frontier-scale language models. You will design core components for fast, reliable, and scalable model training and create tooling that connects research ideas to thousands of GPUs.

The role spans the full ML systems stack, with opportunities to impact data loading, performance profiling, monitoring, reproducibility, and fault-tolerant checkpointing.