Autonomous Platform (Horizon AI)

STN, Inc.; San Francisco (CA); On-site; USD 210,000 - 320,000; Full time

Job summary

Horizon AI is building the operating system for how organizations run themselves. The role seeks a Principal-level engineer to make a fast-moving platform dependable enough for autonomous action and to serve as technical deputy when the Chief Architect is unavailable.

Initial focus is reliability: agents already act on production infrastructure, and we will expand trust only as the dispatch layer becomes robust. The candidate will own orchestration and drive infrastructure improvements.

Qualifications

• Hands-on engineer with senior-level distributed systems experience.
• Experience taking a fast-built system to production dependability: idempotency, circuit breaking, backpressure, and degradation that is observable.
• Proficiency in Python and Java; comfortable with backend services, data infrastructure, and rapid deployment in the same week.
• Judgment about orchestration and scaling grounded in operating systems; ability to reduce novel problems to working code quickly.

Responsibilities

• Fix dispatch reliability with circuit breakers, retry semantics, health-aware routing, and error handling that surfaces failures.
• Add the caching layer, then the event streaming backbone.
• Own the orchestration decision, whether to stay as-is, move to Kubernetes, or something between, and then execute it.
• Introduce infrastructure-as-code and real deployment practice - Refactor toward per-session isolation so concurrent agent runs are safe.
• Own cross-cutting concerns: service scaffolding standards, testing strategy, CI, and observability of our own platform.
• Rotate through Systems and SRE embeds.

Skills

Distributed systems

Production-ready systems

Python

Java

Backend services

Data infrastructure

Deployment

Tools

Kubernetes

Event streaming

CI/CD pipelines

Job description

Autonomous Platform

Level: Principal (MTS 5) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role.

The senior builder who makes a fast-moving platform dependable enough to trust with autonomous action, and who holds the whole architecture in their head. You are also the technical deputy, making calls when the Chief Architect is unavailable.

Your first mandate is reliability. Agents already act on production infrastructure, and we cannot widen what they are trusted with while the dispatch layer is unreliable.

The problems you inherit

• Agent dispatch needs hardening. Routing has no circuit breakers and no fallback path for unreachable agents. This is the hardest blocker to expanding autonomy.

• Errors are absorbed rather than surfaced. Graceful degradation is everywhere, which makes it difficult to distinguish a real failure from a degraded-but-running state.

• No caching layer. Every tool call reads external systems directly, which will rate-limit against systems other teams own as agent volume grows.

• No event streaming. Services talk over synchronous HTTP, so latency cascades between them.

• Single-instance deployment with no horizontal scaling and no infrastructure-as-code.

• Concurrency limits in the agent runtime prevent multiple agent sessions in one process.

You will

• Fix dispatch reliability first: circuit breakers, correct retry semantics, health-aware routing, and error handling that surfaces failures

• Add the caching layer, then the event streaming backbone

• Own the orchestration decision, whether to stay as-is, move to Kubernetes, or something between, and then execute it

• Introduce infrastructure-as-code and real deployment practice - Refactor toward per-session isolation so concurrent agent runs are safe

• Own cross-cutting concerns: service scaffolding standards, testing strategy, CI, and observability of our own platform

• Rotate through Systems and SRE embeds

You have

• A long record of shipping and hardening complex distributed systems as a hands-on engineer at senior level

• Real experience taking a fast-built system to production dependability: idempotency, circuit breaking, backpressure, and degradation that is actually observable

• Range across Python and Java, and comfort in backend services, data infrastructure, and deployment in the same week

• Judgment about orchestration and scaling grounded in operating systems rather than reading about them - The instinct to reduce a novel problem to working code quickly

Nice to have:

Kubernetes at production scale, event streaming, CI at scale