Staff Software Engineer AI Inference Infra Orchestrator

Together AI; San Francisco (CA); On-site; USD 240,000 - 280,000; Full time

Benefits offered by this job

Equity

Health insurance

Competitive benefits

Job summary

Together AI in San Francisco is hiring a Software Engineer to build the systems that treat infrastructure as software, turning racks of GPUs into running inference clusters through a manifest-driven platform. You will design and implement the engines that manifest schema and execute against it, while producing production-grade code and CI/CD pipelines.

You will own the provisioning lifecycle, create declarative APIs for the inference team, and automate repairs and reliability measures, ensuring

Qualifications

• Strong software engineering background in Go, Python, or Rust.
• Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows.
• Experience building software control planes or orchestration systems that model state and reconcile it over time.
• Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub.
• A product mindset. You've built internal platforms or APIs consumed by other engineering teams.
• Nice to have: exposure to bare-metal provisioning and networking fundamentals, or GPU infrastructure.
• Nice to have: experience with GPU cluster software stacks and hyperscaler environments.
• Nice to have: systems programming in Rust or Go.

Responsibilities

• Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery to decommission/RMA.
• Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call.
• Automate self-healing: detect degraded nodes, drain safely, trigger repair or replacement, and reintroduce healthy capacity.
• Own reliability of the pipeline: idempotency, retries, rollback, and drift detection.
• Partner with the inference/ML platform team to encode cluster shapes as abstractions in the platform.
• Engineer it like software: strong typing, automated tests, code reviews, and CI/CD for infrastructure code.

Skills

Go

Python

Rust

Software engineering

CI/CD

Event-driven systems

Tools

Temporal

Cadence

Kafka

NATS

SQS

Kubernetes

Workflow engines

Job description

Together AI in San Francisco is hiring a Software Engineer to build the systems that treat infrastructure as software, turning racks of GPUs into running inference clusters through a manifest-driven platform. You will design and implement the engines that manifest schema and execute against it, while producing production-grade code and CI/CD pipelines.

You will own the provisioning lifecycle, create declarative APIs for the inference team, and automate repairs and reliability measures, ensuring