(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra San Francisco

Together Computer Inc

San Francisco (CA)

On-site

USD 250,000 - 320,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Health insurance
Competitive benefits

Job summary

Together AI, the AI Native Cloud, is seeking a Staff AI Inference Reliability Engineer to own production reliability for GPU inference fleets and the Kubernetes-native control plane. You will triage failures, manage SLOs, and drive automated remediation across a distributed, multi-node environment.

You will work with ML researchers, engineers, and product managers to ship reliable tooling, improve latency and throughput, and reduce operational toil through automation.

Qualifications

  • Experience with production reliability on large-scale distributed or GPU-serving systems.
  • Ability to diagnose InfiniBand/RDMA issues in multi-node environments.
  • Experience with Kubernetes controller/CRD architectures and reconciliation loops.
  • Willingness to engineer GPU infrastructure as software.

Responsibilities

  • Own health of the host lifecycle and fleet-scale provisioning workflows.
  • Lead incident response and translate postmortems into permanent fixes.
  • Extend automated self-healing and add new failure-detection plugins.
  • Collaborate with infra, ML platform, and product teams on reliability.
  • Build visibility tools to identify bottlenecks and instability.

Skills

Production reliability
Kubernetes
RDMA/InfiniBand
GPU infrastructure

Tools

Kubernetes CRD
InfiniBand
RDMA

Job description

(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra
About The Role

We're looking for a software engineer, reliability-minded but still a builder, to own what happens in production to the Kubernetes-native control plane that provisions and runs our GPU inference fleet. The platform team builds the manifest-driven API and controllers that let the inference team declare what they need, a cluster, a model deployment, a capacity change, and get it reconciled underneath. Your job is to keep that promise true in the real world: triaging InfiniBand issues when prefill and decode nodes stall exchanging KV cache over RDMA, diagnosing hero-model and LoRA-serving faults that don't show up in generic health checks, and extending the automated self-healing system so each new failure mode becomes a permanent fix instead of a recurring page.

The core value we're after is keeping the decoupling real even when things break, and keeping the fleet efficient without that efficiency becoming a new source of instability. You'll own SLOs and error budgets for inference serving, lead incident response, and turn what postmortems teach you into actual changes in the controllers and automation, not just updated runbooks. You'll also work the reliability side of capacity efficiency, since a mistuned defragmentation or bin-packing pass is itself a production risk. Strong candidates have hands‑on production‑reliability experience on large-scale distributed or GPU-serving systems, are comfortable troubleshooting InfiniBand/RDMA and Kubernetes controller/CRD-based platforms, and think about GPU infrastructure, and the tools that keep it reliable, as software to be engineered.

A product mindset applies: the on‑call experience, the clarity of an alert, and the judgment built into an automated remediation are things you ship and iterate on.

Responsibilities
  • Operate the provisioning state machine in production: own the health of the host lifecycle pipeline, discovery, inference bring‑up, GPU driver/CUDA validation, health checks, decommission/RMA, triaging stuck or failed state transitions at fleet scale and feeding real failure patterns back into how the states and transitions are designed.
  • Triage and extend automated self‑healing: when the node‑scanning and remediation platform misses something, a new failure mode, an InfiniBand issue in disaggregated prefill/decode serving, a GPU Xid variant, a hero-model or LoRA‑specific fault, diagnose it live, drive the fix to production, and turn it into a new detection/remediation plugin so it's automatic the next time.
  • Own end‑to‑end reliability of the pipeline: hold the SLOs and error budgets for provisioning and fleet health, lead incident response when idempotency, retry, rollback, or drift assumptions break down in practice, and drive the systemic fix, not just the hotfix.
  • Partner with the inference/ML platform team from the production side: be the one who knows how real cluster topologies, interconnects, and scheduling constraints actually fail in the field, and feed those failure modes back as first‑class requirements into the platform's abstractions.
  • Work alongside machine learning researchers, engineers, and product managers to bring our latest technologies into production, enabling advanced research through awesome engineering.
  • Introduce new techniques, tools, and architecture that improve the performance, latency, throughput, and efficiency of our model inference stack.
  • Build tools to give us visibility into our bottlenecks and sources of instability, then design and implement solutions to address the highest-priority issues.
  • Lead reliability cross‑functionally: build coalition across infra, ML platform, and product teams around shared reliability priorities, manage stakeholders through incidents and roadmap trade‑offs, and represent the reliability point of view in planning so it shapes decisions before they ship, not after an outage.
Requirements
  • Experience operating a provisioning or host‑lifecycle pipeline in production, comfortable troubleshooting stuck or failed state transitions (discovery, bring‑up, GPU driver/CUDA validation, health checks, decommission/RMA) at fleet scale, and translating real failure patterns back into improvements to the state machine’s design.
  • Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
  • Hands‑on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi‑node training or disaggregated‑inference context — comfortable reading IB port counters and subnet manager logs, not just application metrics.
  • Familiarity with the operational realities of multi‑tenant model serving (LoRA/adapter‑based serving) and with running differentiated SLOs across model pools of different priority.
  • A demonstrated track record of eliminating operational toil through automation and tooling you built yourself, not just process or documentation.
  • Hands‑on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node‑level health/remediation systems.
  • Comfort with event‑driven architectures, backpressure/circuit‑breaking design, and treating observability (metrics, logs, traces) as something built into a system, not added after an incident.
  • A software‑engineering mindset first: you write and ship production code, and you think about GPU infrastructure as software to be engineered, not a fleet to be manually operated.
About Together AI

Together AI, the AI Native Cloud, is purpose‑built for AI engineers. AI application developers get high‑performance inference that scales reliably, fine‑tuning and reinforcement learning for creating frontier‑level specialized models, and pre‑training at massive scale for fully custom intelligence, all around a marketplace of leading open models that teams can run, adapt, and own. Trusted by Cursor, Decagon, ElevenLabs, Salesforce, and Zoom, Together serves 400+ trillion tokens a month.

Compensation

We offer competitive compensation, startup equity, health insurance and other competitive benefits. The US base salary range for this full‑time position is: $250,000 - $320,000 + equity + benefits. Our salary ranges are determined by location, level and role. Individual compensation will be determined by experience, skills, and job‑related knowledge.

Together AI is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.

As set forth in Together AI’s Equal Employment Opportunity policy,we do not discriminate on the basis of any protected group status under any applicable law.

If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection.As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measurethe effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categoriesis as follows:

A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability.

A "recently separated veteran" means any veteran during the three‑year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.

An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.

An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra
(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra

FDE Times • San Francisco (CA)

On-site
USD 250,000 - 320,000
Startup equity
Health insurance
Competitive benefits
(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure
(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure

Together AI • San Francisco (CA)

On-site
USD 250,000 - 320,000
Startup equity
Health insurance
Competitive benefits
(Senior/Junior AI Inference Reliability Engineer, Compute Infra
(Senior/Junior AI Inference Reliability Engineer, Compute Infra

FDE Times • San Francisco (CA)

On-site
USD 160,000 - 280,000
Equity
Health insurance
Research Engineer, Large-Scale Training
Research Engineer, Large-Scale Training

Together AI • San Francisco (CA)

On-site
USD 200,000 - 290,000
Health insurance
Startup equity
Strategic Account Executive
Strategic Account Executive

Together AI • San Francisco (CA), New York (NY)

Hybrid
USD 300,000 - 370,000
[Junior / Senior / Staff] Software Engineer, Inference / Compute Infrastructure Engineering
[Junior / Senior / Staff] Software Engineer, Inference / Compute Infrastructure Engineering

FDE Times • San Francisco (CA)

On-site
USD 160,000 - 280,000
Startup equity
Health insurance
Equity
Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering
Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering

Together Computer Inc • United States

Remote
USD 180,000 - 240,000
Senior Software Engineer - AI Compute, Together Cloud
Senior Software Engineer - AI Compute, Together Cloud

FDE Times • San Francisco (CA)

Hybrid
USD 220,000 - 270,000
Staff Software Engineer - AI Compute, Together Cloud
Staff Software Engineer - AI Compute, Together Cloud

FDE Times • San Francisco (CA)

Remote
USD 260,000 - 300,000
Remote work flexibility
Sr. Field Marketing & Community Manager
Sr. Field Marketing & Community Manager

Together AI • New York (NY)

On-site
USD 190,000 - 225,000