Site Reliability Engineer - Fleet Command & Control

etched; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Etched is seeking a Site Reliability Engineer to help build the command-and-control layer for a fleet of frontier inference systems. You will define health signals, implement alerting standards, and automate recovery workflows, while coordinating with Supercomputing and Infrastructure for telemetry and rollout.

As one of the first SREs, you will influence incident management, on-call practices, and tooling to keep clusters serving with minimal downtime.

Qualifications

• 2+ years in site reliability, production engineering or similar.
• Experience designing alerting, SLOs and incident management.
• Familiarity with tracing and logs across distributed systems.

Responsibilities

• Define what a healthy inference system looks like with signals across racks, nodes and chips.
• Set a common alerting standard with team-specific SLOs.
• Cut alert noise by correlating alerts and retiring unused pages.
• Automate recovery with runbooks, automated restarts and traffic routing.
• Route events to the right engineer, with context and on-site support for hardware tasks.
• Shape fleet data with telemetry pipelines and OpenTelemetry standards.
• Run incident management with clear severity levels, channels, postmortems and handoffs.
• Improve service reliability through design reviews and canaries.

Skills

Site reliability

Production engineering

Incident management

OpenTelemetry

Troubleshooting

Tools

OpenTelemetry

Monitoring systems

Telemetry tooling

Job description

About Etched

Etched is building hardware for frontier intelligence. We co-design chips, racks, software, and manufacturing to deliver best-in-class throughput and latency across both prefill and decode workloads. Our first products are heavily focused on inference. Backed by hundreds of millions from top-tier investors and staffed by leading engineers, Etched is redefining the infrastructure layer for the fastest growing industry in history.

Job Summary

You'll be one of Etched's first Site Reliability Engineers, building the command-and-control layer for our fleet of frontier inference systems: the alerting, automated recovery and fleet data that keep clusters serving, and the routing that gets each problem to the right expert. Our Supercomputing and Infrastructure teams build the fleet management, rollout and telemetry backbone; but you'll decide which signals define a healthy system, set how we alert and respond, and build agentic hardware health workflows on top of it.

Your job is to design and run a scalable system. Fleet SREs own detection, first response and handoff; domain SREs embedded in each team own the deep product fixes. Over time, most problems should be caught by an alert that measures what users experience, which is then triaged by agents, and sent to an engineer as needed, with the logs, dashboards and recent changes already attached.

Key Responsibilities

• Define what a healthy inference system looks like. In collaboration with software teams building out management and telemetry systems, identify the signals that most clearly show an unhealthy site, rack, node or chip, keep refining them as the fleet changes, and decide which actions (drain, cordon, restart, reroute, roll back) run on their own and which page a person.
• Set a common alerting standard. Define SLOs with each team (latency such as time to first token, error rate, availability), alert on user-facing symptoms, and verify that the metrics behind each alert measure what the alert assumes.
• Cut alert noise. Suppress expected alerts during deploys and maintenance automatically, correlate related alerts into one incident, and retire pages nobody acts on.
• Automate recovery. Turn runbooks into agentic hardware health recovery: service restarts, node drain and traffic shift, spare swaps. Build it with guardrails (scoped permissions, rate limits, blast-radius checks, an audit trail).
• Route every event to the right engineer. Classify events by domain, page the right domain SRE with context attached, and dispatch on-site technicians for physical work such as tray swaps, reseats and leak response.
• Shape the fleet data. Work with Supercomputing and Infrastructure, who build the telemetry pipeline, so it captures the key signals. Set the tracing standard (OpenTelemetry) so every service can be followed end to end, and report uptime and reliability trends that drive engineering and capacity decisions.
• Run operationally excellent incident management. Own severity levels, incident channels, communications, postmortems and corrective-action tracking. Design on-call rotations, handoffs and onboarding so on-call is sustainable.
• Raise the reliability bar for services. Review service designs with their owners for failure modes and graceful degradation before they ship, and partner with Supercomputing on canaries, rolling node-by-node deploys and graceful drain, so updates stop causing downtime.
• Share on-call in a follow-the-sun rotation with SREs in other time zones.

Hands-on debug

• Be the first line of defense. Acknowledge, size the user impact, open the incident, and decide fast between restart, drain, rollback and escalation.
• Find where requests are stuck. Trace a latency or throughput regression through the serving stack (queueing, prefill, decode, handoffs, cache) to the stage that's backing up.
• Triage hardware on live racks. Work through errors: chips dropping off PCIe, firmware panic resets, chip-to-chip link errors, HBM ECC errors and power faults; run recovery tools and decide when a node comes out.
• Correlate across layers. Follow a request end to end with distributed traces (OpenTelemetry), then line up deploys, firmware events, host and BMC logs, network data and facilities telemetry to tell a software bug from a hardware fault from a coolant or power event.
• Check the alert, not just the system. When a page and a dashboard disagree, work out which one is wrong: a capped metric, a stale exporter or a failed scrape.
• Direct hands on site. Walk on-site technicians through reseats, PSU and tray swaps, and leak response with exact steps.
• Hand off cleanly. Leave a timeline, evidence and current hypothesis so the domain SRE or next shift picks up without starting over.
• Close the loop. Turn every manual fix into a runbook, then into tasks that can be executed by agents, so the same page doesn't wake anyone twice.

You may be a good fit if you have (Must-have qualifications)

2+ years in site reliability, production engineering or i