Remote GPU Inference Infra Platform Engineer

Together Computer Inc; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Together AI, the AI Native Cloud, is seeking a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet.

You'll design a manifest-driven API where the inference team declares what they need, and our controllers handle reconciliation and lifecycle management underneath, so the inference team never has to know which serving stack or hardware pool does the work.

Qualifications

• Strong software engineering background in Go, Python, Rust, or similar.
• Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
• Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
• Experience with event-driven systems - designing and building software around message queues, event streams, or pub/sub (e.g., Kafka, NATS, SQS) rather than polling or cron-driven scripts.
• A product mindset. You've built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

Responsibilities

• Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA - as explicit, versioned states and transitions.
• Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call - no ticket, no human in the loop.
• Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
• Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
• Partner with the inference/ML platform team: understand the cluster shapes they need - topology, interconnect, scheduling constraints - and encode them as first-class abstractions in the platform.
• Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code - this is a product, not a collection of Ansible playbooks.

Skills

Go

Python

Rust

Temporal

Cadence

Kubernetes controllers

Kafka

NATS

SQS

Product mindset

Tools

Kubernetes

CI/CD

Linux

Job description

Together AI, the AI Native Cloud, is seeking a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet.

You'll design a manifest-driven API where the inference team declares what they need, and our controllers handle reconciliation and lifecycle management underneath, so the inference team never has to know which serving stack or hardware pool does the work.