Lead AI Systems Engineer for Large-Scale Training

Cerence; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

Cerence is seeking an experienced ML Systems Engineer to design and operate distributed training infrastructure for large neural networks across GPU clusters. You will optimize multi-node, multi-GPU execution and diagnose bottlenecks in compute, memory, and network to maximize throughput and utilization.

Collaborate with research and applied ML teams to productionize large-model training pipelines, improve stability, and implement advanced memory optimizations such as activation checkpointing

Qualifications

• Deep hands-on experience with distributed systems or ML systems.
• Experience running large-scale workloads on GPU clusters.
• Production experience with PyTorch distributed training.
• Strong understanding of parallelism strategies (data, tensor, pipeline).
• Low-level understanding of GPU communication and networking.
• GPU orchestration: Slurm, Kubernetes, Ray, RunAI.
• Communication libraries: NCCL, RDMA, InfiniBand, NVLink.
• Training frameworks: PyTorch Distributed, Megatron-LM, DeepSpeed.
• Memory optimisation: activation checkpointing, ZeRO offload techniques.

Responsibilities

• Distributed Training Infrastructure: Build and optimize GPU cluster orchestration.
• Ensure efficient scheduling, isolation, and fairness across training workloads.
• Communication & Networking optimization.
• Training frameworks: scale large-model training and manage multi-node launches.
• Memory and performance optimization across compute and data paths.
• Collaborate with research and applied ML teams to productionize pipelines.

Skills

Distributed systems

ML systems

GPU clusters

PyTorch Distributed

Megatron-LM

DeepSpeed

NCCL

RDMA

InfiniBand

NVLink

Slurm

Kubernetes

Ray

RunAI

Activation checkpointing

ZeRO

Tools

Slurm

Kubernetes

Ray

RunAI

NCCL

RDMA

InfiniBand

NVLink

PyTorch Distributed

Megatron-LM

DeepSpeed

Job description

Cerence is seeking an experienced ML Systems Engineer to design and operate distributed training infrastructure for large neural networks across GPU clusters. You will optimize multi-node, multi-GPU execution and diagnose bottlenecks in compute, memory, and network to maximize throughput and utilization.

Collaborate with research and applied ML teams to productionize large-model training pipelines, improve stability, and implement advanced memory optimizations such as activation checkpointing