Staff Software Engineer, Self-Service AI Inference Platform

Togetherai; Greater London; Hybrid; GBP 120,000 - 180,000; Full time

Job summary

Together AI is building cutting-edge AI infrastructure. We are seeking a Software Engineer to design and implement manifest-driven systems that provision hardware and turn racks of GPUs into running inference clusters, with a single API to stand up, scale, or tear down workloads.

You will own the lifecycle orchestration, ensure reliability and CI/CD readiness, and work with the ML platform team to tailor cluster shapes to research needs.

Qualifications

• Strong software engineering background in Go, Python, Rust or similar — you write and test real software for a living.
• Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
• Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
• Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub rather than polling or cron-driven scripts.
• A product mindset. You've built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

Responsibilities

• Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery to decommission.
• Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call.
• Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
• Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is dependable.
• Partner with the inference/ML platform team to encode cluster shapes as abstractions in the platform.
• Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code.

Skills

Go

Python

Rust

Tools

Temporal

Cadence

Kubernetes

Kafka

NATS

SQS

Job description

Together AI is building cutting-edge AI infrastructure. We are seeking a Software Engineer to design and implement manifest-driven systems that provision hardware and turn racks of GPUs into running inference clusters, with a single API to stand up, scale, or tear down workloads.

You will own the lifecycle orchestration, ensure reliability and CI/CD readiness, and work with the ML platform team to tailor cluster shapes to research needs.