Agent Felet & Multi Agent Systems (Horizon AI)

STN Inc; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

Horizon AI is seeking a Principal to Distinguished AI role focusing on the reasoning layer for dozens of agents that diagnose fleet problems and decide actions. You will also gate autonomous actions with evidenced-based safety and build the foundation for future agent capabilities.

You will own the architecture and governance of agent orchestration, tool dispatch, and context management, while improving explainability of Agent recommendations to drive trusted outcomes.

Qualifications

• Built and shipped LLM-based agentic systems in production.
• Experience with tool and function calling at scale.
• Strong opinions about evaluation of autonomous agents.
• Experience with multi-agent orchestration and context management.

Responsibilities

• Build the evaluation harness for agent behaviour: scenario suites from real incidents and CI gating.
• Define the autonomy SLO with the Chief Architect and present evidence for moving from recommendation to action.
• Own the agent architecture: orchestration, tool dispatch, task decomposition, escalation, governance boundaries.
• Improve how agents explain their reasoning to support actionable decisions.
• Encode operational runbooks into agent-executable reasoning.

Skills

Agent architecture

Multi-agent orchestration

Python distributed systems

LLM reasoning and explanation

Evaluation design for autonomous AI

Tool and function calling at scale

LLM tracing and online evaluation

Tools

Python

LLM tooling

Job description

Agent Fleet & Multi Agent Systems

Level: Principal to Distinguished (MTS 5 to 6) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role. Own the reasoning layer: dozens of agents that diagnose fleet problems, decide what should happen, and explain why. You also own the thing that does not exist yet and gates everything, which is a way to prove those agents are right often enough to be trusted with more action.

The problems you inherit

• There is no agent-behaviour evaluation. Our existing replay harness tests code-fix regression, not operational decision quality. Building evaluation against real fleet scenarios is greenfield.

• There is no autonomy SLO. No defined, measurable bar an agent must clear before it is trusted with a new class of action. Without one, every expansion of autonomy is a judgment call rather than a decision backed by evidence.

• Multi-agent orchestration is real but fragile. Dispatch and meta-audit exist; reliability does not yet match the ambition.

• The actuation ladder is incomplete. Higher-risk remediations are specified but unbuilt. Designing what agents are permitted to do, and proving they should be, is your work with the Chief Architect.

You will

• Build the evaluation harness for agent behaviour: scenario suites drawn from real incidents, regression gating in CI, and metrics that catch an agent being confidently wrong

• Define the autonomy SLO with the Chief Architect, and own the evidence presented when we propose moving a decision from recommendation to action

• Own the agent architecture: orchestration, tool dispatch, task decomposition, escalation, and the governance boundaries that separate read from write

• Improve how agents explain their reasoning, because an unexplained recommendation does not get acted on

• Continue encoding operational runbooks into agent-executable reasoning

You have

• Built and shipped LLM-based agentic systems in production, not prototypes, including the failure modes: malformed tool output, loops, stuck states, context exhaustion

• Strong and specific opinions about evaluation. This is the part of the job we most need and least have

• Experience with tool and function calling at scale, and with LLM tracing and online evaluation

• Strong systems engineering fundamentals in Python, since agents that touch infrastructure are distributed systems first

• Experience with multi-agent orchestration, handoff, and context management

Nice to have: operational or SRE background, human-in-the-loop system design, retrieval over operational data, experience defining what an autonomous system is allowed to do