Lead, Build Agent Evaluation & Model Benchmarking

ServiceNow; Santa Clara (CA); On-site; USD 166,500 - 291,400; Full time

Benefits offered by this job

Health plans

401(k) plan with company match

ESPP

Matching donations

Flexible time away plan

Family leave programs

Job summary

ServiceNow in Santa Clara, CA is seeking a leader for the Build Agent evaluation framework. You will own eval strategy, roadmap, telemetry, and cross-team quality standards, guiding an 8‑engineer team to deliver scalable, data‑driven improvements.

You will benchmark models, decide on model support, and communicate results to stakeholders. Strong experience with evaluation systems for LLMs and cross‑team accountability is required.

Qualifications

• Direct experience building or leading evaluation systems for LLM-based products, including scoring methodology and benchmark design.
• Ability to drive cross-team accountability and get issues fixed from other teams based on data.
• Working understanding of coding agent architecture: tool use, agent loops, context management.
• 6+ years’ experience with ServiceNow-relevant technologies and advanced coding skills (Java, C++, Ruby, Shell, JavaScript).
• Experience critically evaluating foundation models and distinguishing capability gaps from tuning artifacts.
• Ability to navigate ambiguous priorities with context, risk, and outcomes.
• People management experience at scale (8+ engineers) or managing managers is a plus.

Responsibilities

• Own eval strategy and roadmap: prompt set design, scoring, failure taxonomy, telemetry.
• Coordinate across multiple teams to ensure quality standards and drive fixes.
• Lead model benchmarking and make data-backed recommendations on support decisions.
• Manage daily activities of an 8-engineer team, including staffing and mentoring.
• Address cross-cutting problems where eval data, model capability, and architecture intersect.
• Present eval results and recommendations to stakeholders, including executives.

Skills

Direct experience building evaluation

Cross-team accountability

Coding agent architecture

Java/C++/Ruby/JS expertise

Evaluating foundation models

Ambiguous-priority execution

People management (8+ engineers)

Job description

ServiceNow in Santa Clara, CA is seeking a leader for the Build Agent evaluation framework. You will own eval strategy, roadmap, telemetry, and cross-team quality standards, guiding an 8‑engineer team to deliver scalable, data‑driven improvements.

You will benchmark models, decide on model support, and communicate results to stakeholders. Strong experience with evaluation systems for LLMs and cross‑team accountability is required.