Senior Backend Engineer - AI Inference & Low-Latency

Fdetimes; San Francisco (CA); On-site; USD 200,000 - 290,000; Full time

Benefits offered by this job

Startup equity

Health insurance

Job summary

Together AI invites seasoned engineers to shape the core inference backbone powering frontier models. You will tackle global routing, scaling, and resource allocation across data centers, collaborating with researchers to bring new architectures into production.

Join a team that blends open-source collaboration with industry-leading hardware stacks (H100s, H200s, GB200s), delivering fast, scalable AI inference with equity and competitive benefits.

Qualifications

• 5+ years of experience building large-scale distributed systems and API microservices.
• Strong background in efficiency, scalability, and stability of complex systems.
• Familiarity with low-level OS concepts: multi-threading, memory management, networking.

Responsibilities

• Build and optimize global and local request routing with low-latency load balancing.
• Develop auto-scaling to allocate resources across data centers.
• Design multi-tenant traffic shaping ensuring fair and consistent user experience.
• Trade-offs between latency and throughput for diverse workloads.
• Optimize prefix caching to speed up responses.
• Collaborate with ML researchers to productionize models.
• Profile and optimize system performance to remove bottlenecks.

Skills

Rust

Go

Python

TypeScript

Kubernetes

CUDA

Triton

NCCL

MPI

Education

BS/MS in CS/CE or related field

Tools

CUDA

Triton

NCCL

MPI

Job description

Together AI invites seasoned engineers to shape the core inference backbone powering frontier models. You will tackle global routing, scaling, and resource allocation across data centers, collaborating with researchers to bring new architectures into production.

Join a team that blends open-source collaboration with industry-leading hardware stacks (H100s, H200s, GB200s), delivering fast, scalable AI inference with equity and competitive benefits.