Inference & Workload Optimization Lead (Horizon AI)

STN, Inc.; United States; Remote; USD 180,000 - 280,000; Full time

Job summary

STN, Inc. seeks an Inference & Workload Optimization Lead to own how customer AI workloads run on their committed GPU capacity. You will profile workloads, establish baselines, and build agent-driven optimizations that scale across clusters.

The role involves collaborating with operating teams and advising on GPU allocation between internal and client workloads. Ideal candidates bring deep GPU performance engineering skills, experience with profiling tools (Nsight, DCGM) and distributed

Qualifications

• Deep GPU performance engineering: profiling, memory and bandwidth behaviour, kernel-level understanding.
• Explain performance findings to customers and encode for agents.
• Experience with distributed training/inference at scale and NCCL/tensor/pipeline/data-parallelism.

Responsibilities

• Profile customer training and inference workloads to identify bottlenecks.
• Establish performance baselines per hardware generation (starting with current NVIDIA hardware).
• Turn expertise into agent-executable optimization to scale fixes across clusters.
• Work with committed-capacity customers on workload tuning.
• Own internal model serving as a secondary responsibility including cost and latency-aware routing and model migration.
• Advise on GPU allocation between internal inference and customer workloads.

Skills

GPU performance engineering

Profiling and benchmarking

Distributed training/inference atScale

Explain performance findings to a non-

Tools

Nsight

DCGM

Job description

Inference & Workload Optimization Lead

Level: Principal (MTS 5) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role. Own how well customer AI workloads run on the capacity they have committed to, and build the autonomy that makes that expertise scale past one person’s attention.

This is not a role about serving models to our own agents; that already works. This is about making a client’s training and inference jobs measurably faster on their dedicated cluster, which makes the capacity they bought more valuable.

The problems you inherit

• Optimization is expert-dependent and manual. There is no systematic profiling of customer workloads and no repeatable playbook.

• No performance baselines. We cannot yet say what good looks like per hardware generation, which means we cannot distinguish a badly configured customer job from a well configured one.

• None of it is encoded. Whatever optimization knowledge exists lives in people’s heads rather than in agents that can apply it across every cluster.

• Internal model serving needs maturing, as a secondary duty. Serving works; routing is basic, with no cost or latency awareness.

• GPU capacity for internal inference is negotiated manually against a fleet largely committed to customers.

You will

• Profile customer training and inference workloads and find the real bottlenecks: data loading, collective communication, memory bandwidth, kernel efficiency, batching, parallelism layout

• Establish performance baselines and a definition of good per hardware generation, starting with current-generation NVIDIA hardware

• Turn that expertise into agent-executable optimization, so a fix reaches every cluster rather than the one you happened to look at

• Work with committed-capacity customers, directly or through the operating teams, on workload tuning

• Own internal model serving as a secondary responsibility, including cost and latency-aware routing and model version migration

• Advise on GPU allocation between internal inference and customer workloads

You have

• Deep GPU performance engineering: profiling, memory and bandwidth behaviour, and enough kernel-level understanding to know where time actually goes - Serving stack depth, for example vLLM, SGLang, or TensorRT-LLM, including batching, KV cache management, and quantization trade-offs - Distributed training and inference experience at scale: tensor, pipeline, and data parallelism, and collective communication tuning such as NCCL

• Fluency with profiling tooling, for example Nsight, DCGM, and framework-level profilers

• The ability to explain a performance finding to a customer and to encode it for an agent. These are different skills and this role needs both

Nice to have: CUDA kernel work, open-weight model deployment at scale, benchmark design, prior customer-facing technical experience