Senior Backend Engineer, AI Inference Platform

Togetherai; San Francisco (CA); On-site; USD 160,000 - 250,000; Full time

Benefits offered by this job

Health insurance

Startup equity

Competitive benefits

Job summary

Togetherai in San Francisco seeks an experienced engineer to build and optimize its Inference Platform. You will work with the latest generative AI models and collaborate with researchers to improve system performance and efficiency.

The ideal candidate has a strong background in distributed systems, proficient programming skills, and experience in low-level OS concepts. Competitive salary range from $160,000 to $250,000, along with equity and benefits.

Qualifications

• 5+ years of experience in large-scale, fault-tolerant systems.
• Strong background in efficiency, scalability, and stability.
• Excellent understanding of memory management and networking.
• Expert programming skills in Rust, Go, Python, or TypeScript.
• Familiarity with modern LLMs and generative models.
• Experience with the open source ecosystem around inference.
• Kubernetes experience is a strong plus.
• Familiarity with GPU software stacks is advantageous.

Responsibilities

• Build and optimize global request routing for low-latency load balancing.
• Develop auto-scaling systems across multiple data centers.
• Design systems for traffic shaping and smart rate limiting.
• Engineer trade-offs between latency and throughput.
• Optimize prefix caching to improve response times.
• Collaborate with ML researchers on new model architectures.
• Profile system-level performance and implement optimizations.

Skills

Distributed systems

API microservices

Low-level OS concepts

Rust

Go

Python

TypeScript

Kubernetes

GPU software stacks

Education

Bachelor’s or Master’s degree in Computer Science or equivalent

Tools

CUDA

Triton

NCCL

InfiniBand

Job description

Togetherai in San Francisco seeks an experienced engineer to build and optimize its Inference Platform. You will work with the latest generative AI models and collaborate with researchers to improve system performance and efficiency.

The ideal candidate has a strong background in distributed systems, proficient programming skills, and experience in low-level OS concepts. Competitive salary range from $160,000 to $250,000, along with equity and benefits.