Senior AI Agent Evaluation Engineer

NVIDIA Corporation

Washington (District of Columbia)

On-site

USD 184,000 - 357,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

NVIDIA Corporation is seeking a Senior Software Engineer to build core libraries, evaluation systems, and reusable harness capabilities for AI agents. You will work hands-on across open-source agent harnesses, sandboxed execution, model-provider infrastructure, and benchmarks to improve agent safety, speed, and reliability.

You will diagnose failures from traces and logs, design experiments, and ship improvements with upstream contributions.

Qualifications

  • 8+ years of hands‑on software‑engineering experience with production systems and architecture design leadership.
  • Expert Python skills and working proficiency in Rust, Go, C++, or TypeScript across language/process boundaries.
  • Experience building or extending LLM agents, coding agents, tool‑use loops, model-provider integrations, developer tools, or evaluation infrastructure.
  • Solid understanding of asynchronous execution, subprocesses, containers, sandboxing, callbacks, retries, networking, and distributed compute.
  • Experience crafting reproducible experiments, benchmark methodology, performance investigations, regression gates, or reliability analysis under nondeterministic conditions.
  • A track record of shipping and maintaining open‑source or developer‑facing software with testing, API development, documentation, and code reviews.

Responsibilities

  • Define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmark environments.
  • Build and operate scalable, reproducible evaluation systems spanning CI, scheduled compute, sandboxed execution, artifact provenance, trace capture, and analysis.
  • Diagnose agent failures from traces, logs, tool results, and benchmark artifacts, separating harness, evaluator, environment, inference, and model causes.
  • Design controlled experiments and promotion criteria that distinguish genuine harness improvements from noise, benchmark artifacts, or model-specific gains.
  • Ship focused harness and runtime improvements, including upstream open-source contributions, detectors, regression tests, and maintainable documentation.
  • Partner with Relay, Hermes, model, infrastructure, and research teams to turn recurring optimization needs into reusable runtime, trace, and evaluation capabilities.

Skills

Python
Rust
Go
C++
TypeScript

Education

Bachelor's or Master's degree in CS/AI/Applied Math or related field

Job description

NVIDIA Corporation is seeking a Senior Software Engineer to build core libraries, evaluation systems, and reusable harness capabilities for AI agents. You will work hands-on across open-source agent harnesses, sandboxed execution, model-provider infrastructure, and benchmarks to improve agent safety, speed, and reliability.

You will diagnose failures from traces and logs, design experiments, and ship improvements with upstream contributions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Agent Evaluation Engineer
Senior AI Agent Evaluation Engineer

NVIDIA Corporation • Santa Clara (CA)

On-site
CAD 263,000 - 510,000
Equity
Comprehensive benefits
Senior Software Engineer, Open Agent Harness & Evaluation
Senior Software Engineer, Open Agent Harness & Evaluation

NVIDIA Corporation • California

Hybrid
USD 184,000 - 357,000
Senior AI Agent QA Engineer - Automation & Testing
Senior AI Agent QA Engineer - Automation & Testing

NVIDIA Corporation • United States

On-site
USD 168,000 - 322,000
Equity
Benefits
Senior Software Engineer, Open Harness Engineering
Senior Software Engineer, Open Harness Engineering

NVIDIA Corporation • Santa Clara (CA)

On-site
CAD 263,000 - 510,000
Equity
Comprehensive benefits
Senior AI Evaluation Engineer — Equity & Workflows
Senior AI Evaluation Engineer — Equity & Workflows

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 224,000 - 357,000
Equity and benefits
Senior Open Harness AI Engineer
Senior Open Harness AI Engineer

NVIDIA Corporation • United States

Remote
USD 184,000 - 288,000
Senior AI for Code: Research Engineer (Agents & Tools)
Senior AI for Code: Research Engineer (Agents & Tools)

NVIDIA Corporation • United States

On-site
USD 224,000 - 431,000
Equity
Benefits
Senior Software Engineer, Open Harness Engineering
Senior Software Engineer, Open Harness Engineering

NVIDIA Corporation • Washington

On-site
USD 184,000 - 357,000
Senior AI Safety & Evaluation Engineer
Senior AI Safety & Evaluation Engineer

NVIDIA Corporation • United States

Remote
USD 184,000 - 357,000
Equity
Benefits
Senior AI Systems Architect: Scalable Agent Runtimes
Senior AI Systems Architect: Scalable Agent Runtimes

NVIDIA Corporation • United States

Remote
USD 224,000 - 431,000