Staff Engineer, RL & Post-Training Systems (Remote)

Cohere; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Weekly lunch stipend

Health and dental benefits

RRSP matching / 401K

Parental leave top-up

Education & learning stipend

Paid vacation 6 weeks

Travel budget to other offices

Home office stipend

Job summary

Cohere is seeking a senior software engineer to join our post-training and RL research integration team. You will design high-perf software for training models, build tools to accelerate research, and collaborate with Infrastructure, Efficiency, Serving, and research squads.

This remote-friendly role can be based anywhere within UTC-06:00 to UTC+01:00 or from one of our offices; we value exceptional talent above location, with generous benefits including health, parental leave, and a work-life

Qualifications

• Extremely strong software engineering skills.
• Value test-driven development methods, clean code, and strive to reduce technical debts at all levels.
• Proficiency in Python and related ML frameworks such as JAX, Pytorch and/or XLA/MLIR.
• Experience using and debugging large-scale distributed training strategies (memory/speed profiling).
• [Bonus] Experience with distributed training infrastructures (Kubernetes) and associated frameworks (Ray).
• [Bonus] Hands-on experience with the post-training phase of model training, with a strong emphasis on scalability and performance.
• [Bonus] Experience in ML, LLM and RL academic research.

Responsibilities

• Design and write high-performing and scalable software for training models.
• Develop new tools to support and accelerate research and LLM training.
• Coordinate with other engineering teams (Infrastructure, Efficiency, Serving) and the scientific teams to create a strong post-training ecosystem.
• Craft and implement techniques to improve performance and speed up training cycles (SFT, offline preference, RL).
• Research, implement, and experiment with ideas on our cluster and data infrastructure.
• Collaborate with scientists, engineers, and teams to share knowledge and build cohesive systems.

Skills

Software engineering

Test-driven development

Python ML

Distributed training

LLM/RL research

Performance optimization

Tools

Kubernetes

Ray

JAX

PyTorch

XLA/MLIR

Job description

Cohere is seeking a senior software engineer to join our post-training and RL research integration team. You will design high-perf software for training models, build tools to accelerate research, and collaborate with Infrastructure, Efficiency, Serving, and research squads.

This remote-friendly role can be based anywhere within UTC-06:00 to UTC+01:00 or from one of our offices; we value exceptional talent above location, with generous benefits including health, parental leave, and a work-life