Lead Training Infra Engineer for Scalable AI Models

Cohere; New York (NY); Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP/401K plan

Job summary

Cohere is a leading security-first enterprise AI company focused on building foundation models and end-to-end products for real-world business problems. We welcome candidates for a role that blends research and production, with no remote location restrictions and multiple offices worldwide.

The role emphasizes contributing to training pipelines, shipping frontier models to production, and bridging research with scalable production code.

Qualifications

• Extremely strong software engineering skills.
• Proficiency in Python and ML frameworks such as JAX, PyTorch and XLA/MLIR.
• Experience with distributed training infrastructures (Kubernetes, Slurm) and associated frameworks (Ray)
• Experience using large-scale distributed training strategies.
• Hands on experience on training large model at scale and having contributed to the tooling and/or setup of the training infrastructure
• Bonus: paper at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP).

Responsibilities

• Design and write high-performant and scalable software for training.
• Improve our training setup from an infrastructure and codebase performance standpoint.
• Craft and implement tools to speed up our training cycles and improve the overall efficacy of our training infrastructure
• Research, implement, and experiment with ideas on our supercompute and data infrastructure.
• Learn from and work with the best researchers in the field.

Skills

Software engineering

Python ML

Distributed training

Kubernetes & Slurm

Large-scale training

Research tooling

Tools

JAX

PyTorch

XLA/MLIR

Ray

Kubernetes

Slurm

Job description

Cohere is a leading security-first enterprise AI company focused on building foundation models and end-to-end products for real-world business problems. We welcome candidates for a role that blends research and production, with no remote location restrictions and multiple offices worldwide.

The role emphasizes contributing to training pipelines, shipping frontier models to production, and bridging research with scalable production code.