Site Reliability Engineer - Fleet Command & Control

Etched; San Jose (CA); On-site; USD 150,000 - 240,000; Full time

Benefits offered by this job

Medical benefits

Waiver credit for medical benefits ($)

Housing subsidy $2,500/month for those

Relocation support to San Jose

Wellness benefits

Daily lunch and dinner

Unlimited compute budget

Job summary

Etched is seeking a Site Reliability Engineer to build the command-and-control layer for a fleet of frontier inference systems. You’ll design health signals, automate recovery, and manage incident response to keep clusters serving reliably.

You will define SLO-based alerts, automate health workflows, and triage incidents across teams, coordinating with on-site technicians for hardware and software recovery. A strong focus on Linux, Kubernetes, and IaC will be essential.

Qualifications

• 2+ years in site reliability, production engineering or infrastructure for large-scale production systems.
• Defined alerting standards adopted by multiple teams using Prometheus, Grafana or similar; deep knowledge of SLOs and error budgets.
• Real on-call experience for services with external users; led incidents and written blameless postmortems.
• Proven track record debugging production incidents in unfamiliar systems under time pressure using logs, metrics, traces and core dumps.
• Hands-on proficiency with Linux, containers, Kubernetes and infrastructure as code (Terraform, Ansible or similar).
• Fluency with Linux debugging tools: perf, strace, gdb, dmesg, journalctl, lspci, tcpdump.
• Experience with application and scripting languages such as Python, Go, Rust or similar.
• Comfort with hardware-in-the-loop failures: PCIe device drop-off, ECC errors, power and thermal events.
• Sound judgment about automation scoping and when to elevate to humans.

Responsibilities

• Define signals of system health and decide which automated actions (drain, cordon, restart, reroute, rollback) should run autonomously vs. page engineers.
• Set SLO-based alerting standards across teams and suppress noisy alerts during deployments.
• Automate hardware health recovery workflows with guardrails: restarts, node drain, traffic shift, and spare swaps.
• Triage events and dispatch on-site technicians for physical work; classify by component.
• Debug production incidents end-to-end: trace latency, correlate hardware and software with traces and logs; guide recovery steps.
• Design on-call rotations, incident management processes, severity levels, postmortems and corrective-action tracking; run blameless post-incident reviews.

Skills

Site reliability

Incident management

On-call experience

Linux debugging

Kubernetes

Prometheus

Grafana

SLOs / error budgets

Distributed traces

Automation / IaC

Tools

Terraform

Ansible

OpenTelemetry

Job description

Etched is building custom AI inference hardware and systems. You will be one of their first Site Reliability Engineers, designing the command-and-control layer for their fleet of frontier inference systems - managing alerting, automated recovery, fleet health monitoring, and incident response to keep clusters serving reliably.

What You'll Do

• Define signals of system health and decide which automated actions (drain, cordon, restart, reroute, rollback) execute autonomously versus page engineers
• Set SLO-based alerting standards across teams, suppress expected alerts during deployments, correlate related alerts, and eliminate noise
• Automate hardware health recovery workflows with guardrails: service restarts, node drain, traffic shift, and spare swaps
• Triage and route events to the right domain expert; classify by component and dispatch on-site technicians for physical work
• Debug production incidents end-to-end: trace latency through the serving stack, correlate hardware and software layers using distributed traces and multi-layer logs, and direct on-site technicians through recovery steps
• Design on-call rotations, incident management processes, severity levels, postmortems and corrective-action tracking; run blameless post-incident reviews

What You Need

• 2+ years in site reliability, production engineering or infrastructure for large-scale production systems (8+ years for the San Jose lead seat)
• Defined alerting standards adopted by multiple teams using Prometheus, Grafana or similar; deep knowledge of SLOs, error budgets and alert design
• Real on-call experience for services with external users: led incidents and written blameless postmortems
• Proven track record debugging production incidents in unfamiliar systems under time pressure using logs, metrics, traces and core dumps
• Hands-on proficiency with Linux, containers, Kubernetes and infrastructure as code (Terraform, Ansible or similar)
• Fluency with Linux debugging tools: perf, strace, gdb, dmesg, journalctl, lspci, tcpdump
• Experience with application and scripting languages such as Python, Go, Rust or similar
• Comfort with hardware-in-the-loop failures: PCIe device drop-off, ECC errors, power and thermal events, failures crossing software and physical layers
• Sound judgment about automation scoping and when to elevate to humans

Nice to Have

• Operated GPU, accelerator or HPC fleets
• LLM inference serving experience: prefill/decode, KV cache, batching, time-to-first-token latency metrics
• Network debugging across switches, load balancers, edge controls and incident traffic rerouting
• Out-of-band management: BMC, Redfish, IPMI, and fleet-wide firmware rollouts
• Data center telemetry: power distribution, liquid cooling, CDUs, DCIM
• PagerDuty or similar incident tooling: event correlation and automated remediation
• Experience founding or scaling an SRE function at a startup

• Medical, dental, and vision packages with generous premium coverage
• $500/month credit for waiving medical benefits
• $2,500/month housing subsidy for those within walking distance of office
• relocation support to San Jose
• wellness benefits (fitness, mental health)
• daily lunch and dinner
• unlimited compute budget subject to ROI justification