ML Evaluation Engineer — Agent Trajectory Scoring

ServiceNow; Mountain View (CA); On-site; USD 114,000 - 176,000; Full time

Benefits offered by this job

Health plans

401(k) plan with company match

Employee stock purchase plan (ESPP)

Donations matching

Flexible time away

Job summary

ServiceNow is seeking a role focused on building and operating the evaluation framework for its AI agents, anchored in Moveworks technology. You will own the calibration, scoring, and reliability of the multi-tenant eval harness, enabling scalable, repeatable experiments across enterprise simulations.

This position emphasizes back-end systems, data pipelines, and observability. The role involves Python/Go, distributed systems, and orchestration across complex enterprise environments.

Qualifications

• Experience with large-scale distributed systems and reproducible results.
• Familiarity with orchestration runtimes and high-concurrency job systems.
• Hands-on work with observability tooling (OpenTelemetry).
• Proficiency in Python or Go; ability to design backend infra handling real traffic at scale.
• Experience with data-intensive pipelines and schema evolution.
• Experience designing and operating systems using gRPC/protobuf.

Responsibilities

• Architect the evaluation and scoring pipelines for agent-based systems.
• Develop end-to-end eval environments, simulators, and harnesses.
• Ensure reliability, observability, and reproducibility of eval runs.
• Collaborate across teams to integrate eval signals into release gates.

Skills

Distributed systems

Orchestration workflows

OpenTelemetry observability

Python/Go

Data-intensive pipelines

gRPC/protobuf

Tools

Temporal

Airflow

Argo

OpenTelemetry

Job description

ServiceNow is seeking a role focused on building and operating the evaluation framework for its AI agents, anchored in Moveworks technology. You will own the calibration, scoring, and reliability of the multi-tenant eval harness, enabling scalable, repeatable experiments across enterprise simulations.

This position emphasizes back-end systems, data pipelines, and observability. The role involves Python/Go, distributed systems, and orchestration across complex enterprise environments.