Site Reliability Engineer — Fleet AI Inference & Automation

LinuxCareers; San Jose (CA); Sur place; USD 150 000 - 240 000; Plein temps

Avantages offerts par ce poste

Medical benefits

Waiver credit for medical benefits ($)

Housing subsidy $2,500/month for those

Relocation support to San Jose

Wellness benefits

Daily lunch and dinner

Unlimited compute budget

Résumé du poste

Etched is seeking a Site Reliability Engineer to build the command-and-control layer for a fleet of frontier inference systems. You’ll design health signals, automate recovery, and manage incident response to keep clusters serving reliably.

You will define SLO-based alerts, automate health workflows, and triage incidents across teams, coordinating with on-site technicians for hardware and software recovery. A strong focus on Linux, Kubernetes, and IaC will be essential.

Qualifications

• 2+ years in site reliability, production engineering or infrastructure for large-scale production systems.
• Defined alerting standards adopted by multiple teams using Prometheus, Grafana or similar; deep knowledge of SLOs and error budgets.
• Real on-call experience for services with external users; led incidents and written blameless postmortems.
• Proven track record debugging production incidents in unfamiliar systems under time pressure using logs, metrics, traces and core dumps.
• Hands-on proficiency with Linux, containers, Kubernetes and infrastructure as code (Terraform, Ansible or similar).
• Fluency with Linux debugging tools: perf, strace, gdb, dmesg, journalctl, lspci, tcpdump.
• Experience with application and scripting languages such as Python, Go, Rust or similar.
• Comfort with hardware-in-the-loop failures: PCIe device drop-off, ECC errors, power and thermal events.
• Sound judgment about automation scoping and when to elevate to humans.

Responsabilités

• Define signals of system health and decide which automated actions (drain, cordon, restart, reroute, rollback) should run autonomously vs. page engineers.
• Set SLO-based alerting standards across teams and suppress noisy alerts during deployments.
• Automate hardware health recovery workflows with guardrails: restarts, node drain, traffic shift, and spare swaps.
• Triage events and dispatch on-site technicians for physical work; classify by component.
• Debug production incidents end-to-end: trace latency, correlate hardware and software with traces and logs; guide recovery steps.
• Design on-call rotations, incident management processes, severity levels, postmortems and corrective-action tracking; run blameless post-incident reviews.

Connaissances

Site reliability

Incident management

On-call experience

Linux debugging

Kubernetes

Prometheus

Grafana

SLOs / error budgets

Distributed traces

Automation / IaC

Outils

Terraform

Ansible

OpenTelemetry

Description du poste

Etched is seeking a Site Reliability Engineer to build the command-and-control layer for a fleet of frontier inference systems. You’ll design health signals, automate recovery, and manage incident response to keep clusters serving reliably.

You will define SLO-based alerts, automate health workflows, and triage incidents across teams, coordinating with on-site technicians for hardware and software recovery. A strong focus on Linux, Kubernetes, and IaC will be essential.