AI Systems SRE: Observability, Resilience & Canary

Tech Mirrors; Mountain View (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Tech Mirrors is seeking a Senior SRE specialized in AI platforms to own observability, metrics, and incident response across high-traffic production systems. You will build and extend O11Y pipelines, instrument AI workloads, and define robust SLOs with cost-aware monitoring.

Expertise in Go or Python, Kubernetes and AWS, and hands-on experience with modern ML monitoring tools is required. Join a blameless, incident-driven team focused on reliability and scalable AI infrastructure.

Qualifications

• 7+ years SRE/production engineering on Tier-0/Tier-1, high-traffic systems.
• Strong Go or Python - build role: tooling, automation, instrumentation.
• Has built observability stacks, not just consumed them: Prometheus/Grafana, OpenTelemetry, or equivalent at scale, including cardinality and cost control.
• Production LLM/ML monitoring: Langfuse, Arize, WhyLabs, or homegrown - token/cost tracking, drift and quality metrics.
• Working fluency in AI-system failure modes: nondeterminism, provider limits and outages, context-window overflow, agent loops, cache poisoning.
• Kubernetes + AWS operational depth - debugs across cluster, mesh, and gateway layers.
• Structured incident-command and postmortem experience.

Responsibilities

• Build and extend O11Y pipelines: metrics, distributed tracing, logging, SLO/SLI definitions and dashboards consumed platform-wide.
• Instrument AI systems in production: token usage and cost, latency-per-inference, tool-call success rates, response-quality signals, agent loop detection, runaway-cost alerting.
• Extend tracing across agentic flows - planner → executor → retrieval → tool calls - spanning GenOS, AI Gateway, and MCP Gateway.
• Define SLOs where "available" includes response quality and tool-call success, not just HTTP 200s.
• On-call and incident command for AI platform surfaces: model/provider degradation, Bedrock/Gemini failover, semantic-cache issues, prompt-injection events - blameless postmortems with tracked remediations.
• Own the SRE side of the progressive-delivery seam: canary analysis, automated rollback decisioning, chaos/resilience testing, blast-radius controls - DevOps builds the pipeline; you define the gates.
• Build AIOps agents in the Traffic Agent pattern: intelligent triage, anomaly detection, auto-remediation with hard guardrails.
• Drive UX Availability (R30) and Foundations Capabilities (R30); land network, O11Y, and cloud cost savings (tracked YTD KPIs).

Skills

Go

Python

Observability

Prometheus

Grafana

OpenTelemetry

LLM monitoring

Tools

Prometheus/Grafana/OpenTelemetry

Langfuse

Arize

WhyLabs

Job description

Tech Mirrors is seeking a Senior SRE specialized in AI platforms to own observability, metrics, and incident response across high-traffic production systems. You will build and extend O11Y pipelines, instrument AI workloads, and define robust SLOs with cost-aware monitoring.

Expertise in Go or Python, Kubernetes and AWS, and hands-on experience with modern ML monitoring tools is required. Join a blameless, incident-driven team focused on reliability and scalable AI infrastructure.