Principal AI Ops Engineer

Crux AI; Palo Alto (CA); Hybrid; USD 190,000 - 270,000; Full time

Benefits offered by this job

Health, dental, and vision coverage

401(k) Plan with 4% company match

Hybrid schedule: four days in-office,

Life insurance and disability

Full suite of other optional benefits

Job summary

Crux AI is building the systems that let AI agents, not people, run the TPU fleet. You will write telemetry pipelines, AI agents, and automation to detect, triage, and remediate failures across facilities and the control plane.

You will turn runbooks into code and implement guardrails, SLOs, and incident response, contributing to a fast-paced, reliability-focused engineering culture.

Qualifications

• 10+ years in software or SRE roles overseeing large-scale infrastructure.
• Proven experience building AI-driven operations systems like event correlation or auto-remediation.
• Strong Python or Go development and hands-on observability (Prometheus/OpenTelemetry).

Responsibilities

• Build and own telemetry and event pipelines tying power, cooling, TPU hosts, interconnect, and control plane.
• Write AI agents to triage alerts, runbooks, drain/requeue workloads, and open hardware repair/RMA requests.
• Turn runbooks into code with signals, alerts, automated responses, and owners for critical failures.
• Build guardrails with blast-radius limits, approval gates, and tested rollback paths.
• Define and instrument SLOs and error budgets with Compute, Network Engineering, and Facility Operations.
• Take part in on-call and incident response, driving postmortems that ship fixes.
• Measure automation share and drive higher zero-human-touch operation per release.

Skills

SRE

Production engineering

Python/Go

Tools

Prometheus

OpenTelemetry

Kubernetes

Job description

Built to set the gold standard for integrated AI infrastructure

Crux AI is a newly formed, U.S.-based integrated AI infrastructure company created to remove the physical and operational constraints on consequential AI ambitions. Crux brings together power, high-density data centers, TPU silicon, networking, orchestration software, and ongoing operations as one integrated system.

Crux is being capitalized to plan every layer together, develop each one to demanding standards, and operate the whole system with efficiency and reliability. That gives hyperscalers, frontier AI labs, sovereign customers, enterprises, and AI-native companies greater freedom to pursue the AI they are here to create.

WHAT YOU’LL DO

You will build the systems that let AI agents, not people, do most of the work of running Crux’s TPU fleet. Reporting to our Director, AI Ops, you will be one of the first engineers on the team, writing the telemetry pipelines, correlation logic, and agents that detect, triage, and remediate failures across facilities, TPU hosts, the network, and the control plane before a customer notices. You will turn every runbook into code and every repeat incident into automation. This is a hands-on engineering seat: you write and ship the software, then operate it in production.

In this role, you will:

• Build and own the telemetry and event pipeline that ties together power, cooling, TPU hosts, interconnect, and the control plane, with the correlation to tell a failing pump from a failing chip from a failing job.

• Write AI agents that triage alerts, execute runbooks, drain and requeue workloads, and open hardware repair and RMA requests with Google, with every action logged, reversible, and measured.

• Turn runbooks into code, so that every critical failure mode for the first build maps to a signal, an alert, an automated response, and an owner.

• Build the guardrails that keep autonomous actions safe: blast-radius limits, approval gates for high-risk changes, and tested rollback paths.

• Define and instrument SLOs and error budgets with Compute, Network Engineering, and Facility Operations, and build the views the team reviews every week.

• Take part in on-call and incident response, and drive postmortems that end in a fix that ships, not an action item that ages.

• Measure the share of operational work that closes with zero human touch, and drive that number up release over release.

SIGNALS OF SUCCESS

After 60 days in this role:

• Gained a complete understanding of the first-build architecture, the failure modes that matter most, and the tooling in place today, and shipped a first working agent against a real, repeat failure.

• Stood up the first version of the unified telemetry pipeline for the first build.

• Adapted to a fast-paced environment, triaging appropriately and continuously improving our function.

After 6 months:

• Taken the first Stage 1 cluster into production with automated detection and remediation in place for its top failure modes.

• Shipped agents the team relies on every day, with a measured share of incidents resolved without human intervention.

• Built the guardrails and audit trail that let the team trust autonomous actions in production.

After 1 year:

• Running automation across a multi-site fleet where most repeat operational work executes autonomously, with measurable gains in availability, recovery time, and cost per chip.

• Built a library of agents and runbooks-as-code that every new site inherits on day one.

• Become a reason strong SRE and operations engineers choose Crux.

EXPERIENCES, ATTRIBUTES AND MINDSET THAT INDICATE A GOOD MATCH

Experiences

• 10+ years in software engineering, SRE, or production engineering for large-scale infrastructure, including operating a fleet across many sites against hard SLAs.

• Built and shipped AI-driven operations systems in production, such as event correlation, anomaly detection, auto-remediation, or agentic triage, with measurable results.

• Strong software skills in Python or Go, and hands‑on experience with observability stacks (for example Prometheus or OpenTelemetry), Kubernetes, and large-scale data pipelines.

Attributes

• You are outcome driven and hands‑on. You read the telemetry and write the agent yourself. This is a builder’s seat, not a supervisory one.

• You are close enough to the hardware to be useful. You understand how power, cooling, accelerators, and networks fail, not just how software does.

• You have a high tolerance for ambiguity. You absorb ambiguity and navigate complexity rather than passing it along.

Mindset

• You are skeptical by default. “90% automated” is a hypothesis until it survives a bad night at a new site. You build the system that makes the next incident boring.

• You deliver reliability through discipline, not heroics. You would rather fix the cause once than page someone twice.

• You are AI-agentic first. Fluent with AI agents, or committed to becoming so quickly, you embed them as first principles in how you and your team work, defaulting to agentic workflows before adding headcount or process.

Nice to have (Preferred, not required):

• SRE or production engineering at a hyperscaler (Google, Meta, Microsoft, AWS) or a neocloud (CoreWeave, Nebius, Lambda, Crusoe).

• Experience operating TPU or GPU clusters for large training and inference workloads, including hardware health, repair workflows, and fleet telemetry.

• Experience with data center infrastructure telemetry such as BMS, DCIM, or power and cooling systems.

• Experience building LLM-based agents with tool use, evaluations, and safety controls in production.

• Experience with predictive maintenance, ML-based anomaly detection, or failure prediction at fleet scale.

Salary Range Information

The annual salary range for this position has been estimated based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description.

About Crux

• We offer generous base, bonus and additional incentive based compensation

• Health, dental, and vision coverage for you and your dependents

• Company-paid life insurance and disability

• Full suite of other optional benefits

• 401(k) Plan with 4% company match

• Hybrid schedule offering four days in-office collaboration paired with one remote workday for focused, individual work