AI Benchmarking & Automation Engineer

ServiceNow; San Diego (CA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Magnit Global benefits

Job summary

ServiceNow in San Diego, CA, is seeking an engineer to help build and scale tooling that measures AI-powered software development tools. You will develop evaluation harnesses, automate benchmark runs, and ensure reproducible results across teams.

The role emphasizes measurement quality over automation, with collaboration across engineering and data teams to document methods and findings for technical and leadership audiences.

Qualifications

• Strong software engineering background with automation experience.
• Experience with Git and Docker.
• Experience with APIs, CI/CD, and engineering workflows.
• Familiarity with AI, LLM, or agent evaluation and benchmarking.
• Ability to troubleshoot and analyze results.

Responsibilities

• Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks.
• Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time.
• Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.
• Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.
• Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and find ways to make the workflows more reliable and more automated.
• Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences.

Skills

Software engineering

Automation

Developer tooling

Git

Docker

APIs

CI/CD pipelines

AI/LLM evaluation

Troubleshooting

Tools

Docker

CI/CD tooling

Job description

ServiceNow in San Diego, CA, is seeking an engineer to help build and scale tooling that measures AI-powered software development tools. You will develop evaluation harnesses, automate benchmark runs, and ensure reproducible results across teams.

The role emphasizes measurement quality over automation, with collaboration across engineering and data teams to document methods and findings for technical and leadership audiences.