Senior ML Engineer: Agentic Evaluation & Reward Modeling

Servicenow; Santa Clara (CA); Hybrid; USD 150,000 - 230,000; Full time

Job summary

ServiceNow seeks an experienced ML-focused engineer to own the evaluation framework for AI agents. You will design judges, calibration rubrics, and confidence-aware scoring, working across deterministic validators and LLM judgments to improve agent performance in enterprise workflows.

The role emphasizes building a production-grade evaluation platform, collaborating with annotation teams, and advancing self-learning for agent harnesses with rigorous measurement and ownership at startup pace.

Qualifications

• 5+ years in applied ML, data science, or ML-adjacent engineering, with a track record of work that shipped and got used.
• Experience turning subjective human judgement into a measurement that holds up — one that other models can act on.
• Strong Python, and the discipline to ship production-grade code rather than notebooks.

Responsibilities

• Design and calibrate judges and evaluation rubrics for agent trajectories.
• Build and maintain an evaluation platform with deterministic validators and LLM-based judgments.
• Develop a calibration loop with human-labeled trajectories and collaborate with annotation teams.

Skills

Applied ML experience

Python programming

Measurement/ evaluation design

LLM experience

Technical communication

Job description

ServiceNow seeks an experienced ML-focused engineer to own the evaluation framework for AI agents. You will design judges, calibration rubrics, and confidence-aware scoring, working across deterministic validators and LLM judgments to improve agent performance in enterprise workflows.

The role emphasizes building a production-grade evaluation platform, collaborating with annotation teams, and advancing self-learning for agent harnesses with rigorous measurement and ownership at startup pace.