Platform Reliability (Horizon AI)

STN, Inc.; United States; À distance; USD 180 000 - 240 000; Plein temps

Résumé du poste

STN, Inc. is seeking a Staff Platform Reliability Engineer to make Horizon AI’s software dependable and scalable, working with the Principal Engineer to improve resilience.

You will build a caching layer, enable event streaming, and harden dispatch with health checks, circuit breakers, retries, and observability across services. Proficiency in Python and Kubernetes and a track record of delivering reliability improvements are required.

Qualifications

• Need production experience building reliable distributed systems.
• Proven ability to reduce latency and failure rates.
• Experience implementing caching, queuing, and fault-tolerant patterns.

Responsabilités

• Build caching layer to decouple agents from external systems.
• Implement event streaming to avoid synchronous HTTP chains.
• Harden dispatch with health checks, circuit breakers, retries and observability.
• Develop platform observability including metrics and tracing.
• Contribute to orchestration migration and infrastructure as code.
• Improve deployment practices: canary releases and rollback strategies.

Connaissances

Distributed systems

Caching

Message queues

Circuit breakers

Backpressure

Timeouts

Observability

Kubernetes

Python

Outils

Kubernetes

Python

Docker

Description du poste

Platform Reliability

Level: Staff (MTS 4) · Reports to: Chief AI Officer, later the Senior Engineering Manager

The role. Make our own platform dependable and scalable, working with the Principal Engineer. This is reliability engineering for Horizon AI’s software rather than for the GPU fleet, though the two meet when our agents are the ones being trusted with fleet actions.

You will

• Build the caching layer that stops every agent tool call from hitting external systems directly

• Implement event streaming so services stop depending on synchronous HTTP chains

• Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation

• Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on

• Contribute to the orchestration migration and to infrastructure-as-code

• Improve deployment practice: canary, traffic shifting, and real rollback

You have

• Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency

• Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration

• Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes

• A record of shipping reliability improvements with measurable before and after

Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services