Inference Platform Backend Engineer (Equity & Benefits)

Together; San Francisco (CA); On-site; USD 160,000 - 250,000; Full time

Benefits offered by this job

Equity

Health insurance

Competitive compensation

Job summary

Together AI is hiring a Senior Backend Engineer for its Inference Platform in San Francisco. You’ll shape the core inference backbone, optimize global routing, and build auto-scaling for multi-tenant workloads across data centers.

You’ll work with researchers to productionize frontier models and contribute to open-source projects for faster, cheaper AI at scale. The role demands strong distributed systems, low-level OS knowledge, and expertise in Rust/Go/Python/TypeScript, plus experience with

Qualifications

• 5+ years of experience building large-scale, fault-tolerant distributed systems and API microservices.

Responsibilities

• Build and optimize global and local request routing with low-latency load balancing.
• Develop auto-scaling systems to allocate resources while meeting SLOs across many data centers.
• Design multi-tenant traffic shaping with fair resource allocation and regulated request handling.
• Engineer latency-vs-throughput trade-offs for diverse workloads.
• Optimize prefix caching to speed up model responses.
• Collaborate with ML researchers to productionize new model architectures.
• Profiling system-wide performance to identify bottlenecks and improvements.

Skills

Distributed systems

API microservices

Low-level OS concepts

Rust/Go/Python/TypeScript

Kubernetes

GPU software stacks

Open-source inference tools

Education

Bachelor’s or Master’s in Computer Science/Engineering

Tools

Kubernetes

CUDA

NCCL

Triton

NVLink

MPI

Job description

Together AI is hiring a Senior Backend Engineer for its Inference Platform in San Francisco. You’ll shape the core inference backbone, optimize global routing, and build auto-scaling for multi-tenant workloads across data centers.

You’ll work with researchers to productionize frontier models and contribute to open-source projects for faster, cheaper AI at scale. The role demands strong distributed systems, low-level OS knowledge, and expertise in Rust/Go/Python/TypeScript, plus experience with