Platform Reliability (Horizon AI)

STN, Inc.; San Francisco (CA); Sur place; USD 180 000 - 240 000; Plein temps

Résumé du poste

Horizon AI in San Francisco seeks a Staff-level reliability engineer to make the Horizon AI platform dependable and scalable. You will work with the Principal Engineer to build resilient systems for the software platform, distinct from the GPU fleet but aligned where both touch.

You will design and implement caching, event streaming, robust health checks, and observability, contributing to orchestration migration and IaC, and advancing canary deployments and rollback capabilities to reduce risk

Qualifications

• Must have deep experience building reliable distributed systems
• Strong skills in caching, queuing, and backpressure
• Experience with event streaming and observability
• Proficient in Python and containerized environments (Kubernetes)
• Exposure to orchestration and IaC practices
• Record of shipping reliability improvements with measurable results

Responsabilités

• Build the caching layer to prevent agent calls hitting external systems
• Implement event streaming to avoid synchronous HTTP chains
• Harden agent dispatch with health checks, circuit breakers, retry logic, and good observability
• Develop service-level metrics, tracing, and alerting across the platform
• Contribute to orchestration migration and infrastructure-as-code efforts
• Improve deployment practice with canary, traffic shifting, and rollback strategies

Description du poste

Platform Reliability

Level: Staff (MTS 4) · Reports to: Chief AI Officer, later the Senior Engineering Manager

The role. Make our own platform dependable and scalable, working with the Principal Engineer. This is reliability engineering for Horizon AI’s software rather than for the GPU fleet, though the two meet when our agents are the ones being trusted with fleet actions.

You will

• Build the caching layer that stops every agent tool call from hitting external systems directly

• Implement event streaming so services stop depending on synchronous HTTP chains

• Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation

• Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on

• Contribute to the orchestration migration and to infrastructure-as-code

• Improve deployment practice: canary, traffic shifting, and real rollback

You have

• Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency

• Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration

• Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes

• A record of shipping reliability improvements with measurable before and after

Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services