Staff Platform Reliability Engineer — Scale Distributed AI

STN, Inc.; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Horizon AI in San Francisco seeks a Staff-level reliability engineer to make the Horizon AI platform dependable and scalable. You will work with the Principal Engineer to build resilient systems for the software platform, distinct from the GPU fleet but aligned where both touch.

You will design and implement caching, event streaming, robust health checks, and observability, contributing to orchestration migration and IaC, and advancing canary deployments and rollback capabilities to reduce risk

Qualifications

• Must have deep experience building reliable distributed systems
• Strong skills in caching, queuing, and backpressure
• Experience with event streaming and observability
• Proficient in Python and containerized environments (Kubernetes)
• Exposure to orchestration and IaC practices
• Record of shipping reliability improvements with measurable results

Responsibilities

• Build the caching layer to prevent agent calls hitting external systems
• Implement event streaming to avoid synchronous HTTP chains
• Harden agent dispatch with health checks, circuit breakers, retry logic, and good observability
• Develop service-level metrics, tracing, and alerting across the platform
• Contribute to orchestration migration and infrastructure-as-code efforts
• Improve deployment practice with canary, traffic shifting, and rollback strategies

Job description

Horizon AI in San Francisco seeks a Staff-level reliability engineer to make the Horizon AI platform dependable and scalable. You will work with the Principal Engineer to build resilient systems for the software platform, distinct from the GPU fleet but aligned where both touch.

You will design and implement caching, event streaming, robust health checks, and observability, contributing to orchestration migration and IaC, and advancing canary deployments and rollback capabilities to reduce risk