LLM Evaluation Engineer – AI Agent Platforms

Servicenow; Santa Clara (CA); On-site; USD 126,000 - 195,000; Full time

Benefits offered by this job

Equity

Health plans

401(k) Plan with company match

ESPP

Flexible time away

Family leave

Job summary

ServiceNow is seeking an expert in AI evaluation to advance its Build Agent team, the AI coding assistant for the platform. You will work on evaluation infrastructure, scoring, and failure analysis to ensure high-quality model performance across ServiceNow metadata types and workflows.

Responsibilities include large-scale eval orchestration and improving agent observability and tracing. A track record in LLM evaluation and model benchmarking is required; you will influence release quality and

Qualifications

• Direct experience building evaluation systems for LLM-based products, including scoring methods and benchmarks.
• Understanding of coding agent architecture: tools, loops, context management, scaffolding tradeoffs.
• Experience evaluating foundation models and distinguishing real capability from tuning artifacts.
• Ability to handle ambiguous priorities with incomplete eval data.

Responsibilities

• Eval orchestration at scale
• Agent observability and tracing

Skills

LLM evaluation systems

Agent architecture understanding

Foundation models evaluation

Ambiguity handling

Job description

ServiceNow is seeking an expert in AI evaluation to advance its Build Agent team, the AI coding assistant for the platform. You will work on evaluation infrastructure, scoring, and failure analysis to ensure high-quality model performance across ServiceNow metadata types and workflows.

Responsibilities include large-scale eval orchestration and improving agent observability and tracing. A track record in LLM evaluation and model benchmarking is required; you will influence release quality and