Platform Reliability (Horizon AI)

STN Inc; Oregon; Hybrid; USD 140,000 - 190,000; Full time

Job summary

STN, Inc. is seeking a Platform Reliability Staff to design and implement robust distributed systems.

This role focuses on building a caching layer to shield tools from external systems, enabling event streaming to remove synchronous chains, and hardening dispatch with health checks, circuit breakers, and observability. You will drive reliability improvements across deployment, orchestration, and IaC practices.

Qualifications

• Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency.
• Comfort in Python and containerized environments, with exposure to Kubernetes or equivalent orchestration.
• Experience with event streaming and observability: metrics, tracing, and alerting on symptoms, not causes.
• Track record of shipping reliability improvements with measurable before/after results.

Responsibilities

• Build the caching layer to prevent every agent tool call from hitting external systems directly.
• Implement event streaming so services stop relying on synchronous chains.
• Harden agent dispatch with health checks, circuit breakers, retries, and observability to distinguish failure from degradation.
• Build observability of the platform: service-level metrics, tracing across services, and actionable alerts.
• Contribute to orchestration migration and infrastructure-as-code efforts.
• Improve deployment practices: canary releases, traffic shifting, and safe rollbacks.

Skills

Distributed systems

Caching

Event streaming

Python

Containers

Kubernetes

Observability

Reliability work

Tools

Terraform

CI/CD

Job description

Platform ReliabilityLevel: Staff (MTS 4) · Reports to: Chief AI Officer, later the Senior Engineering ManagerThe role.

You will

• Build the caching layer that stops every agent tool call from hitting external systems directly
• Implement event streaming so services stop depending on synchronous chains
• Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation
• Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on
• Contribute to the orchestration migration and to infrastructure-as-code
• Improve deployment practice: canary, traffic shifting, and real rollback

You have

• Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency
• Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration
• Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes
• A record of shipping reliability improvements with measurable before and after

Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services