Platform Reliability Lead - Scale, Observability & Canary

STN, Inc.; United States; Remote; USD 140.000 - 190.000; Vollzeit

Zusammenfassung

STN, Inc. is seeking a Staff Platform Reliability engineer to make Horizon AI’s software platform dependable and scalable. You will work with the Principal Engineer to design and implement reliability features across caching, event streaming, and observability.

You will lead the reliability improvements for distributed services using Python and Kubernetes, applying canaries, traffic shifts, and robust rollback strategies to ensure resilient deployments.

Qualifikationen

• Production experience making distributed systems reliable (caching, queuing, circuit breaking).
• Comfort with Python and containerized environments (Kubernetes).
• Experience with event streaming and the failure modes each option brings – Observability depth: metrics, tracing, alerting.
• A record of shipping reliability improvements with measurable before and after.

Aufgaben

• Build the caching layer that stops every agent tool call from hitting external systems directly.
• Implement event streaming so services stop depending on synchronous HTTP chains.
• Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation.
• Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on.
• Contribute to the orchestration migration and to infrastructure-as-code.
• Improve deployment practice: canary, traffic shifting, and real rollback.

Kenntnisse

Distributed systems

Caching

Event streaming

Observability

Canary deployment

Tools

Python

Kubernetes

Jobbeschreibung

STN, Inc. is seeking a Staff Platform Reliability engineer to make Horizon AI’s software platform dependable and scalable. You will work with the Principal Engineer to design and implement reliability features across caching, event streaming, and observability.

You will lead the reliability improvements for distributed services using Python and Kubernetes, applying canaries, traffic shifts, and robust rollback strategies to ensure resilient deployments.