Senior AI Ops Engineer: Autonomous Infra & Reliability

Crux, Inc.; Palo Alto (CA); Hybrid; USD 190,000 - 270,000; Full time

Benefits offered by this job

Health, dental, and vision coverage

401(k) Plan with 4% company match

Hybrid schedule: four in-office days +

Job summary

Crux, Inc. is building the next generation of AI infrastructure in the United States. As an early SRE/Production Engineer, you will write telemetry pipelines, AI agents for triage, and automation to detect and remediate issues across TPU fleets and the control plane.

You will turn runbooks into code, establish safety gates and rollback paths, and define SLOs with operations teams. The role emphasizes hands-on engineering, reliability, and scalable operations in a hybrid Palo Alto environment.

Qualifications

• 10+ years in software engineering, SRE, or production engineering for large-scale infrastructure.
• Built and shipped AI-driven operations systems in production (e.g., event correlation, anomaly detection, auto-remediation).
• Strong software skills in Python or Go, observability stacks, Kubernetes, and large-scale data pipelines.

Responsibilities

• Build and own telemetry and event pipelines tying power, cooling, TPU hosts, interconnect, and the control plane.
• Write AI agents to triage alerts, runbooks, and requeue workloads with logged, reversible actions.
• Turn runbooks into code with automated responses and owner mapping for critical failures.
• Build guardrails with blast-radius limits, approval gates, and tested rollback paths.
• Define and instrument SLOs and error budgets, and create team-wide views.
• Participate in on-call, incident response, and postmortems to ship fixes.
• Increase the share of operational work closed with zero human touch across releases.

Skills

Python

Go

Observability

Kubernetes

Large-scale data pipelines

Education

Bachelor's degree in Computer Science

Tools

Prometheus

OpenTelemetry

Terraform

Job description

Crux, Inc. is building the next generation of AI infrastructure in the United States. As an early SRE/Production Engineer, you will write telemetry pipelines, AI agents for triage, and automation to detect and remediate issues across TPU fleets and the control plane.

You will turn runbooks into code, establish safety gates and rollback paths, and define SLOs with operations teams. The role emphasizes hands-on engineering, reliability, and scalable operations in a hybrid Palo Alto environment.