Staff Engineer, Model Efficiency & LLM Inference

Visa Hunt; New York (NY); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP matching

Parental leave top-up

Education stipend

Vacation and offsite

Home office stipend

Job summary

Cohere is hiring for a senior engineer focused on building reliable ML systems and optimizing LLM inference performance. You will work across the inference stack to reduce latency and increase throughput while collaborating with modeling and systems teams.

We offer remote-friendly roles with distributed teams, offices in multiple cities, and extensive perks including health benefits, parental leave, and learning stipends.

Qualifications

• 5+ years of experience writing high-performance, production-quality code.
• Strong programming skills in C++ or Python (Rust/Go also welcome).
• Experience working with large language models and familiarity with the LLM inference ecosystem (e.g., vLLM, SGLang, etc.).
• Ability to diagnose and resolve performance bottlenecks across the model execution stack.
• A strong bias for action — you ship fast, measure impact, and iterate.

Responsibilities

• Improve core performance metrics by optimizing model execution.
• Collaborate with modeling and systems teams to measure and ship improvements.
• Build expertise in GPU/CUDA optimizations and MoE strategies.

Skills

High-performance coding

C++

Python

LLM inference

Performance optimization

Tools

C++

Python

vLLM

SGLang

Job description

Cohere is hiring for a senior engineer focused on building reliable ML systems and optimizing LLM inference performance. You will work across the inference stack to reduce latency and increase throughput while collaborating with modeling and systems teams.

We offer remote-friendly roles with distributed teams, offices in multiple cities, and extensive perks including health benefits, parental leave, and learning stipends.