Staff Software Engineer — Scalable Inference Infra

Cohere; New York (NY); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP matching / 401K

Parental leave top-up

Enrichment benefits

6 weeks vacation

Travel budget to other offices

$500 home office stipend

Job summary

Cohere is seeking a senior staff engineer to join the Model Serving team. You will deploy, operate, and optimize Cohere's AI platform delivering large language models via API endpoints, ensuring low latency, high throughput, and high availability.

You will collaborate with cross-functional teams to meet deployment needs, design scalable distributed systems with Kubernetes and GPUs, and leverage cloud platforms for multi-cloud serving.

Qualifications

• 5+ years of engineering experience running production infrastructure at a large scale.
• Experience designing large, highly available distributed systems with Kubernetes and GPU workloads.
• Experience with Kubernetes dev and production coding and support.
• Experience with GCP, Azure, AWS, OCI, multi-cloud on-prem / hybrid serving.
• Experience in designing, deploying, supporting, and troubleshooting Linux-based environments.
• Experience in compute/storage/network resource and cost management.
• Strong collaboration and troubleshooting skills for mission-critical systems.
• The grit and adaptability to solve complex technical challenges.

Responsibilities

• Develop, deploy, and operate the AI platform delivering Cohere's large language models via API endpoints.
• Collaborate with multiple teams to optimize deployment and performance.
• Interface with customers to create customized deployments for their needs.

Skills

Distributed systems

Kubernetes

GPU workloads

Golang/C++

Cloud platforms

Linux troubleshooting

Collaboration

Problem solving

Tools

Kubernetes

Golang

C++

Linux

GCP

AWS

Azure

Job description

Cohere is seeking a senior staff engineer to join the Model Serving team. You will deploy, operate, and optimize Cohere's AI platform delivering large language models via API endpoints, ensuring low latency, high throughput, and high availability.

You will collaborate with cross-functional teams to meet deployment needs, design scalable distributed systems with Kubernetes and GPUs, and leverage cloud platforms for multi-cloud serving.