Agent Evaluation and Self-Improvement Engineer

NVIDIA Corporation; Santa Clara (CA); On-site; USD 224,000 - 431,000; Full time

Job summary

NVIDIA Corporation in Santa Clara, CA seeks a principal engineer to define evaluation strategy for agents and enterprise-scale systems. You will design how eval results guide self-improvement across skills, plugins, and harness configurations, with sandboxed tests and human sign-off.

Lead cross-team evaluation efforts, mine traces for failures and patterns, and ensure trustworthy improvements. Strong emphasis on experiment design and senior IC leadership.

Qualifications

• BS or MS in Computer Science, Engineering, or related field (or equivalent experience).
• 12+ years building large-scale systems, including evaluation systems for ML or agents and data pipelines.
• Hands-on experience building and evaluating agents on an open or closed agent harness.

Responsibilities

• Build evaluation that works across harnesses with automated checks for regressions.
• Mine production traces for failures, usage patterns, and performance hot spots.
• Turn data into eval cases and improvement signals.
• Develop self-improvement loop for evolving harnesses with sandboxed evals.
• Ensure only measurable gains ship, with human sign-off and limits on agent changes.
• Create realistic enterprise environments for safe long-running tasks.
• Measure multi-agent swarms and track permissions across hops.

Skills

Python

Go

Rust

Experiment design

Senior IC leadership

Education

BS or MS in Computer Science or related

Job description

Autonomous agents now do real work across NVIDIA, and every piece underneath them keeps changing: models, harnesses, tools, skills, and plugins. Evaluation is how a team knows whether a change made an agent better or quietly made it worse. We're looking for a principal engineer to set evaluation strategy at every layer, from individual skills and MCP servers to whole agents and products, and to design how agents use those results to improve themselves. Agents can already rewrite the skills and plugins they run on, then test whether the change helped. Making that loop trustworthy is the hard part. What does it take for an agent to get better on its own, and prove the improvement is real? We're building a system that answers that question for every agent at NVIDIA. Come build it with us!

What you'll be doing:

• Build evaluation that works the same across harnesses, with automated checks on every change that catch regressions before users do.
• Mine production traces for failures, new usage patterns, missing skills and tools, cost and latency hot spots, and successful runs.
• Turn them into eval cases and improvement signals.
• Evaluate the eval datasets themselves: coverage of real usage, duplicate cases, and how closely they match production.
• Build the self-improvement loop for evolving harnesses: agents propose changes to their own skills, plugins, and harness configuration, and sandboxed evals test each one.
• Only measurable gains ship, with human sign-off and limits agents can't change.
• Create realistic enterprise environments: sandboxed replicas of issue trackers, code repositories, chat, and internal APIs, filled with synthetic or anonymized data, where agents safely run long, end-to-end tasks.
• Measure multi-agent swarms as a team: how agents split work, whether handoffs succeed, and whether permissions hold at every hop.
• Record how long each task takes and how many tokens it uses.
• Keep evals honest: calibrate model and agent judges against human labels.
• Catch agents that cheat, such as working backward from expected answers or reaching data and tools they shouldn't access.

What we need to see:

• BS or MS in Computer Science, Engineering, or related field (or equivalent experience) 12+ years building large-scale systems, including evaluation systems for ML or agents and the data pipelines that feed them
• Hands-on experience building and evaluating agents on an open or closed agent harness
• Solid grounding in experiment design: statistical significance, variance across repeated runs, and A/B testing
• Experience setting technical direction across teams as a senior individual contributor
• Proficiency in Python, Go, Rust, or similar

Ways to stand out from the crowd:

• Built self-improving or self-evolving agent systems that get better from their own eval results
• Shipped an agent benchmark, eval framework, or simulated test environment adopted by other teams
• Detected reward hacking or eval gaming, or built oversight for systems that modify themselves
• Built domain evaluations, such as for chip design or hardware verification

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 224,000 USD - 356,500 USD for Level 5, and 272,000 USD - 431,250 USD for Level 6. You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until October 20, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

NVIDIA pioneered accelerated computing. Today, our AI infrastructure powers global intelligence, transforming every industry.

Learn more about NVIDIA.