Senior AI Systems Engineer, Evaluation & Orchestration

Servicenow; Mountain View (CA); On-site; USD 161,000 - 274,000; Full time

Benefits offered by this job

Health plans

401(k) with company match

Employee stock purchase plan (ESPP)

Matching donations

Flexible time away

Job summary

ServiceNow seeks a senior backend/infrastructure engineer to build and operate evaluation and observability tooling for its enterprise AI agent platform. You will design runtimes for multi-turn agent scenarios, implement calibration against human labels, and drive scalable, reproducible experiments across production-scale datasets.

The role emphasizes Python/Go expertise, OpenTelemetry tracing, and building non-deterministic systems that yield meaningful evaluation signals.

Qualifications

• 5+ years building production backend or infrastructure systems
• Strong in Python or Go (ideally both)
• Experience designing and operating systems that handle real traffic at scale
• Comfort making a non-deterministic system measurable
• Comfort with observability tools and tracing (OpenTelemetry)

Responsibilities

• Build the judgement layer of the agent evaluation platform with rubrics and calibration against human labels
• Own the eval orchestration at scale: run multi-turn agent scenarios, transcripts, and final state collection
• Consolidate evals into a single orchestration service with a single source of truth
• Lead observability efforts: OpenTelemetry-native tracing and data models for agent trajectories
• Design and operate stateful simulation environments that mimic enterprise systems

Skills

Python

Go

OpenTelemetry

Distributed systems

Orchestration

Observability

Async programming

gRPC/protobuf

Tools

Temporal

Airflow

Argo

OpenTelemetry

Job description

ServiceNow seeks a senior backend/infrastructure engineer to build and operate evaluation and observability tooling for its enterprise AI agent platform. You will design runtimes for multi-turn agent scenarios, implement calibration against human labels, and drive scalable, reproducible experiments across production-scale datasets.

The role emphasizes Python/Go expertise, OpenTelemetry tracing, and building non-deterministic systems that yield meaningful evaluation signals.