Senior AI Agent Evaluation Engineer

NVIDIA Corporation

Santa Clara (CA)

On-site

CAD 263,000 - 510,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Comprehensive benefits

Job summary

NVIDIA Corporation is seeking a Senior Software Engineer to help build core libraries, evaluation systems, and reusable harness capabilities for safer, faster agent-based software. You will work hands-on across open-source agent harnesses, sandboxed execution, and model-provider infrastructure to improve agent reliability and performance.

The role requires expert Python and proficiency in Rust/Go/C++/TypeScript, with experience in LLM agents, tool-use loops, and evaluation infrastructure.

Qualifications

  • Bachelor's or Master's degree in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
  • 8+ years of hands-on software-engineering experience with production systems; architecture design leadership.
  • Expert Python skills and working proficiency in Rust, Go, C++, or TypeScript, across language boundaries.
  • Experience building or extending LLM agents, coding agents, tool-use loops, model-provider integrations, or evaluation infrastructure.
  • Solid understanding of asynchronous execution, subprocesses, containers, sandboxing, networking, and distributed compute.
  • Experience crafting reproducible experiments, benchmark methodology, performance investigations, or reliability analysis in nondeterministic conditions.
  • Record of shipping/open-source or developer-facing software with testing, API development, documentation, and code reviews.

Responsibilities

  • Define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmark environments.
  • Build and operate scalable, reproducible evaluation systems spanning CI, scheduled compute, sandboxed execution, provenance, trace capture, and analysis.
  • Diagnose agent failures from traces, logs, tool results, and benchmark artifacts across harness, evaluator, environment, inference, and model causes.
  • Design controlled experiments and promotion criteria distinguishing genuine harness improvements from noise or model-specific gains.
  • Ship focused harness and runtime improvements including upstream contributions, detectors, regression tests, and documentation.
  • Partner with Relay, Hermes, model, infrastructure, and research teams to turn recurring optimization needs into reusable runtime, trace, and evaluation capabilities.

Skills

Python
Rust
Go
C++
TypeScript

Education

Bachelor's or Master's in Computer Science or related field

Tools

Open-source agent harnesses
CI systems

Job description

NVIDIA Corporation is seeking a Senior Software Engineer to help build core libraries, evaluation systems, and reusable harness capabilities for safer, faster agent-based software. You will work hands-on across open-source agent harnesses, sandboxed execution, and model-provider infrastructure to improve agent reliability and performance.

The role requires expert Python and proficiency in Rust/Go/C++/TypeScript, with experience in LLM agents, tool-use loops, and evaluation infrastructure.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, Open Agent Harness & Evaluation
Senior Software Engineer, Open Agent Harness & Evaluation

NVIDIA Corporation • California

Hybrid
USD 184,000 - 357,000
Senior AI Systems Architect – Agent Runtimes
Senior AI Systems Architect – Agent Runtimes

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 224,000 - 431,000
Equity
Benefits
Senior Software Engineer, Open Harness Engineering
Senior Software Engineer, Open Harness Engineering

NVIDIA Corporation • Santa Clara (CA)

On-site
CAD 263,000 - 510,000
Equity
Comprehensive benefits
Senior AI Systems Architect: Scalable Agent Runtimes
Senior AI Systems Architect: Scalable Agent Runtimes

NVIDIA Corporation • United States

Remote
USD 224,000 - 431,000
Senior AI Systems Engineer: Agentic CUDA & GPUs, Equity
Senior AI Systems Engineer: Agentic CUDA & GPUs, Equity

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 152,000 - 242,000
Senior AI Platform Engineer - Evaluator & Agentic Systems
Senior AI Platform Engineer - Evaluator & Agentic Systems

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Senior AI Safety Engineer - Equity & Agentic Automation
Senior AI Safety Engineer - Equity & Agentic Automation

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Equity
Benefits package
Competitive salary
Senior Software Engineer, Open Harness Engineering
Senior Software Engineer, Open Harness Engineering

NVIDIA Corporation • California

Hybrid
USD 184,000 - 357,000
Senior AI Safety Engineer — Agentic Automation & Safe Models
Senior AI Safety Engineer — Agentic Automation & Safe Models

Nvidia Corporation in • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Senior AI Evaluation Engineer — Equity & Workflows
Senior AI Evaluation Engineer — Equity & Workflows

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 224,000 - 357,000
Equity and benefits