Senior Backend Engineer — Global AI Inference at Scale

Together Computer Inc; Oregon; Hybrid; USD 200,000 - 290,000; Full time

Benefits offered by this job

Equity

Health insurance

Competitive benefits

Job summary

Together AI is hiring for a senior systems engineer to design and optimize the core inference backbone powering frontier AI models. You will tackle global routing, auto-scaling, and multi-tenant traffic shaping to deliver ultra-low latency and high throughput.

Collaborate with ML researchers and open-source projects to push efficiency and scalability at scale. We seek 5+ years in building large distributed systems, with strong OS knowledge and expertise in Rust/Go/Python/TypeScript.

Qualifications

• 5+ years of demonstrated experience building large-scale, fault-tolerant distributed systems and API microservices.
• Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems.
• Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance.
• Expert-level programming in Rust, Go, Python, or TypeScript.
• Knowledge of modern LLMs and how they are served in production is a plus.
• Experience with the open source ecosystem around inference; familiarity with SGLang, vLLM, or NVIDIA Dynamo is handy.
• Experience with Kubernetes or container orchestration is a strong plus.
• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience.

Responsibilities

• Build and optimize global and local request routing, ensuring low-latency load balancing across data centers and model engine pods.
• Develop auto-scaling systems to dynamically allocate resources and meet strict SLOs across dozens of data centers.
• Design systems for multi-tenant traffic shaping, tuning resource allocation and request handling for fairness and consistent experience.
• Engineer trade-offs between latency and throughput to serve diverse workloads efficiently.
• Collaborate with ML researchers to bring new model architectures into production at scale.
• Continuously profile and analyze system-level performance to identify bottlenecks and implement optimizations.

Skills

Distributed systems

API microservices

Rust

Go

Python

TypeScript

Education

Bachelor's degree in CS/CE or related field

Master's degree in CS/CE or related field

Tools

Kubernetes

CUDA

NVIDIA Dynamo

vLLM

SGLang

Job description

Together AI is hiring for a senior systems engineer to design and optimize the core inference backbone powering frontier AI models. You will tackle global routing, auto-scaling, and multi-tenant traffic shaping to deliver ultra-low latency and high throughput.

Collaborate with ML researchers and open-source projects to push efficiency and scalability at scale. We seek 5+ years in building large distributed systems, with strong OS knowledge and expertise in Rust/Go/Python/TypeScript.