Lead Training Infra Engineer for Scalable AI Models

Cohere

New York (NY)

Remote

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Lunch stipend
Health and dental benefits
RRSP/401K plan

Job summary

Cohere is a leading security-first enterprise AI company focused on building foundation models and end-to-end products for real-world business problems. We welcome candidates for a role that blends research and production, with no remote location restrictions and multiple offices worldwide.

The role emphasizes contributing to training pipelines, shipping frontier models to production, and bridging research with scalable production code.

Qualifications

  • Extremely strong software engineering skills.
  • Proficiency in Python and ML frameworks such as JAX, PyTorch and XLA/MLIR.
  • Experience with distributed training infrastructures (Kubernetes, Slurm) and associated frameworks (Ray)
  • Experience using large-scale distributed training strategies.
  • Hands on experience on training large model at scale and having contributed to the tooling and/or setup of the training infrastructure
  • Bonus: paper at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP).

Responsibilities

  • Design and write high-performant and scalable software for training.
  • Improve our training setup from an infrastructure and codebase performance standpoint.
  • Craft and implement tools to speed up our training cycles and improve the overall efficacy of our training infrastructure
  • Research, implement, and experiment with ideas on our supercompute and data infrastructure.
  • Learn from and work with the best researchers in the field.

Skills

Software engineering
Python ML
Distributed training
Kubernetes & Slurm
Large-scale training
Research tooling

Tools

JAX
PyTorch
XLA/MLIR
Ray
Kubernetes
Slurm

Job description

Cohere is a leading security-first enterprise AI company focused on building foundation models and end-to-end products for real-world business problems. We welcome candidates for a role that blends research and production, with no remote location restrictions and multiple offices worldwide.

The role emphasizes contributing to training pipelines, shipping frontier models to production, and bridging research with scalable production code.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Engineer - AI Systems & Post-Training RL
Staff Engineer - AI Systems & Post-Training RL

Cohere • New York (NY)

Remote
USD 150,000 - 230,000
Lunch stipend and health benefits
Dental benefits
RRSP/401K/Pension
+6
Remote Staff AI Engineer — Post-Training to Production
Remote Staff AI Engineer — Post-Training to Production

Cohere • United States

Remote
USD 175,000 - 380,000
Weekly lunch stipend
Health and dental benefits
RRSP matching / 401K / Pension scheme
+4
Staff ML Engineer: Post-Training & Production
Staff ML Engineer: Post-Training & Production

Cohere • New York (NY)

Remote
USD 150,000 - 210,000
Weekly lunch stipend
Health and dental benefits
RRSP matching / 401K
+5
Research Engineer, Scalable Foundation Model Training
Research Engineer, Scalable Foundation Model Training

Together Computer Inc • San Francisco (CA)

On-site
USD 200,000 - 290,000
Health insurance
Startup equity
Staff AI Platform Engineer — Scalable ML Infra (Hybrid)
Staff AI Platform Engineer — Scalable ML Infra (Hybrid)

Linkedin3 • Mountain View (CA)

Hybrid
USD 210,000 - 270,000
Principal AI Infra Engineer – Agentic Platforms
Principal AI Infra Engineer – Agentic Platforms

Cohere • United States

Remote
USD 140,000 - 220,000
Weekly lunch stipend $75/£75 or local?
Full health and dental benefits
RRSP/401K and pension
+4
ML Engineer: Build Scalable AI Training Pipelines
ML Engineer: Build Scalable AI Training Pipelines

Inference • San Francisco (CA)

Hybrid
USD 220,000 - 320,000
Equity
Comprehensive benefits
Large compute budget
Senior Systems Engineer for Scalable AI Training Infra
Senior Systems Engineer for Scalable AI Training Infra

River AI • Palo Alto (CA)

On-site
USD 200,000 - 420,000
Health insurance
Dental insurance
Vision insurance
+3
AI Training Infrastructure Engineer for Scalable ML
AI Training Infrastructure Engineer for Scalable ML

Skild AI • San Mateo (CA)

On-site
USD 100,000 - 300,000
AI Training Infrastructure Engineer
AI Training Infrastructure Engineer

designworkstalent • United States

Hybrid
USD 150,000 - 210,000
Medical, dental, vision insurance
401(k) with company match
Paid holidays