Senior AI Agent Evaluation Engineer

NVIDIA Gruppe; California (MO); On-site; USD 184,000 - 357,000; Full time

Job summary

NVIDIA seeks a senior software engineer to define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmarks. You will build scalable evaluation systems, diagnose failures from traces and logs, and design experiments to distinguish real harness improvements from noise.

Responsibilities include shipping harness/runtime improvements, contributing to open-source, and collaborating with model, infrastructure, and research teams to

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Applied Math, or related field or equivalent experience.
• 8+ years of hands-on software-engineering experience with production systems and architecture design leadership.
• Expert Python skills and working proficiency in Rust, Go, C++, or TypeScript; able to debug across language boundaries.
• Experience building or extending LLM agents, tool-use loops, model-provider integrations, or evaluation infrastructure.
• Solid understanding of asynchronous execution, subprocesses, containers, sandboxing, callbacks, retries, networking, and distributed compute.
• Experience crafting reproducible experiments, benchmark methodology, performance investigations, or reliability analysis under nondeterministic conditions.
• Record of shipping and maintaining open-source or developer-facing software with testing, API development, documentation, and code reviews.

Responsibilities

• Define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmark environments.
• Build and operate scalable, reproducible evaluation systems spanning CI, scheduled compute, sandboxed execution, artifact provenance, trace capture, and analysis.
• Diagnose agent failures from traces, logs, tool results, and benchmark artifacts, distinguishing harness, evaluator, environment, inference, and model causes.
• Design controlled experiments and promotion criteria that distinguish genuine harness improvements from noise or model-specific gains.
• Ship harness and runtime improvements, including upstream open-source contributions, detectors, regression tests, and maintainable documentation.
• Partner with Relay, Hermes, model, infrastructure, and research teams to turn recurring optimization needs into reusable runtime, trace, and evaluation capabilities.

Skills

Python

Rust

Go

C++

TypeScript

LLM agents

asynchronous execution

distributed compute

debugging across boundaries

open-source software

Education

Bachelor’s or Master’s in Computer Science/AI/Applied Math

Tools

OpenTelemetry

OpenInference

AgentBench

Job description

NVIDIA seeks a senior software engineer to define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmarks. You will build scalable evaluation systems, diagnose failures from traces and logs, and design experiments to distinguish real harness improvements from noise.

Responsibilities include shipping harness/runtime improvements, contributing to open-source, and collaborating with model, infrastructure, and research teams to