Senior Backend Engineer, Inference Platform — Global Low-Latency

Together Computer Inc; United States; Remote; USD 160,000 - 250,000; Full time

Benefits offered by this job

Equity

Health insurance

Competitive benefits

Job summary

Together AI is seeking a backend systems engineer to build the core inference backbone for frontier models, with a focus on low-latency routing, autoscaling, and multi-tenant scheduling across data centers.

You will work with research teams and open-source projects like SGLang, vLLM, and NVIDIA Dynamo, delivering scalable, efficient infrastructure and contributing to production-grade systems on GPU clusters.

Qualifications

• 5+ years of experience building large-scale distributed systems.
• Expert in API design and microservices with fault tolerance.
• Strong knowledge of OS concepts: multithreading, memory, networking, storage.
• Proficient in Rust, Go, Python, or TypeScript.
• Experience with Kubernetes and GPU software stacks is a plus.

Responsibilities

• Build and optimize global and local request routing with low latency.
• Develop auto-scaling systems across many data centers.
• Design multi-tenant traffic shaping for fair resource allocation.
• Trade latency vs throughput for diverse workloads.
• Optimize prefix caching to speed up responses.
• Collaborate with ML researchers to productionize new models.
• Profile and optimize system performance and remove bottlenecks.

Tools

Rust

Go

Python

TypeScript

Kubernetes

CUDA

Triton

NCCL

Job description

Together AI is seeking a backend systems engineer to build the core inference backbone for frontier models, with a focus on low-latency routing, autoscaling, and multi-tenant scheduling across data centers.

You will work with research teams and open-source projects like SGLang, vLLM, and NVIDIA Dynamo, delivering scalable, efficient infrastructure and contributing to production-grade systems on GPU clusters.