Production GPU Inference Reliability Engineer

Fdetimes; San Francisco (CA); On-site; USD 250,000 - 320,000; Full time

Benefits offered by this job

Startup equity

Health insurance

Competitive benefits

Job summary

Together AI, the AI Native Cloud, seeks a software engineer focused on reliability to own the production health of our GPU inference fleet and Kubernetes-native control plane. You will triage issues, extend automated self-healing, and drive end-to-end reliability across provisioning, fleet health, and capacity decisions.

Ideal candidates have hands-on production reliability experience in large-scale distributed or GPU-serving systems, with comfort troubleshooting InfiniBand/RDMA and Kubernetes

Qualifications

• Experience operating a provisioning or host-lifecycle pipeline in production, comfortable troubleshooting stuck or failed state transitions (discovery, bring-up, GPU driver/CUDA validation, health checks, decommission/RMA) at fleet scale, and translating real failure patterns back into improvements to the state machine’s design.
• Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
• Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi-node training or disaggregated-inference context — comfortable reading IB port counters and subnet manager logs, not just application metrics.
• Familiarity with the operational realities of multi-tenant model serving (LoRA/adapter-based serving) and with running differentiated SLOs across model pools of different priority.
• A demonstrated track record of eliminating operational toil through automation and tooling you built yourself, not just process or documentation.
• Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
• Comfort with event-driven architectures, backpressure/circuit-breaking design, and treating observability (metrics, logs, traces) as something built into a system, not added after an incident.
• A software-engineering mindset first: you write and ship production code, and you think about GPU infrastructure as software to be engineered, not a fleet to be manually operated.

Responsibilities

• Operate the provisioning state machine in production: own the health of the host lifecycle pipeline, discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA, triaging stuck or failed state transitions at fleet scale and feeding real failure patterns back into how the states and transitions are designed.
• Triage and extend automated self-healing: when the node-scanning and remediation platform misses something, a new failure mode, an InfiniBand issue in disaggregated prefill/decode serving, a GPU Xid variant, a hero-model or LoRA-specific fault, diagnose it live, drive the fix to production, and turn it into a new detection/remediation plugin so it's automatic the next time.
• Own end-to-end reliability of the pipeline: hold the SLOs and error budgets for provisioning and fleet health, lead incident response when idempotency, retry, rollback, or drift assumptions break down in practice, and drive the systemic fix, not just the hotfix.
• Partner with the inference/ML platform team from the production side: be the one who knows how real cluster topologies, interconnects, and scheduling constraints actually fail in the field, and feed those failure modes back as first-class requirements into the platform's abstractions.
• Work alongside machine learning researchers, engineers, and product managers to bring our latest technologies into production, enabling advanced research through awesome engineering.
• Introduce new techniques, tools, and architecture that improve the performance, latency, throughput, and efficiency of our model inference stack.
• Build tools to give us visibility into our bottlenecks and sources of instability, then design and implement solutions to address the highest-priority issues.
• Lead reliability cross-functionally: build coalition across infra, ML platform, and product teams around shared reliability priorities, manage stakeholders through incidents and roadmap trade-offs, and represent the reliability point of view in planning so it shapes decisions before they ship, not after an outage.

Job description

Together AI, the AI Native Cloud, seeks a software engineer focused on reliability to own the production health of our GPU inference fleet and Kubernetes-native control plane. You will triage issues, extend automated self-healing, and drive end-to-end reliability across provisioning, fleet health, and capacity decisions.

Ideal candidates have hands-on production reliability experience in large-scale distributed or GPU-serving systems, with comfort troubleshooting InfiniBand/RDMA and Kubernetes