Agent Felet & Multi Agent Systems (Horizon AI)

STN, Inc.; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Horizon AI seeks a Principal to Distinguished (MTS 5–6) to own the reasoning layer for a fleet of autonomous agents operating on STN’s GPU infrastructure in SF. You will design how agents diagnose problems, decide actions, and explain decisions, while gating actions with an evidence-based framework.

You will build evaluation harnesses, set autonomy SLOs with the Chief Architect, and shape the governance around tool use, escalation, and write permissions across distributed systems.

Qualifications

• Built and shipped LLM-based agentic systems in production, including failure modes and debugging.
• Strong opinions about evaluation and measurable success criteria in autonomous systems.
• Experience with tool and function calling at scale, and with LLM tracing and online evaluation.

Responsibilities

• Own the reasoning layer: dozens of agents that diagnose fleet issues and decide actions.
• Own the gating mechanism to prove agent decisions before action.
• Define and implement evaluation harnesses for agent behaviour in real incidents.

Skills

LLM-based agent systems

Python distributed systems

Multi-agent orchestration

LLM tracing

Job description

Agent Fleet & Multi Agent Systems

Level: Principal to Distinguished (MTS 5 to 6) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role. Own the reasoning layer: dozens of agents that diagnose fleet problems, decide what should happen, and explain why. You also own the thing that does not exist yet and gates everything, which is a way to prove those agents are right often enough to be trusted with more action.

The problems you inherit

• There is no agent-behaviour evaluation. Our existing replay harness tests code-fix regression, not operational decision quality. Building evaluation against real fleet scenarios is greenfield.

• There is no autonomy SLO. No defined, measurable bar an agent must clear before it is trusted with a new class of action. Without one, every expansion of autonomy is a judgment call rather than a decision backed by evidence.

• Multi-agent orchestration is real but fragile. Dispatch and meta-audit exist; reliability does not yet match the ambition.

• The actuation ladder is incomplete. Higher-risk remediations are specified but unbuilt. Designing what agents are permitted to do, and proving they should be, is your work with the Chief Architect.

You will

• Build the evaluation harness for agent behaviour: scenario suites drawn from real incidents, regression gating in CI, and metrics that catch an agent being confidently wrong

• Define the autonomy SLO with the Chief Architect, and own the evidence presented when we propose moving a decision from recommendation to action

• Own the agent architecture: orchestration, tool dispatch, task decomposition, escalation, and the governance boundaries that separate read from write

• Improve how agents explain their reasoning, because an unexplained recommendation does not get acted on

• Continue encoding operational runbooks into agent-executable reasoning

You have

• Built and shipped LLM-based agentic systems in production, not prototypes, including the failure modes: malformed tool output, loops, stuck states, context exhaustion

• Strong and specific opinions about evaluation. This is the part of the job we most need and least have

• Experience with tool and function calling at scale, and with LLM tracing and online evaluation

• Strong systems engineering fundamentals in Python, since agents that touch infrastructure are distributed systems first

• Experience with multi-agent orchestration, handoff, and context management

Nice to have: operational or SRE background, human-in-the-loop system design, retrieval over operational data, experience defining what an autonomous system is allowed to do