Senior AI Agent Evaluation Engineer

Socket.dev; Town of Texas (WI); Hybrid; USD 184,000 - 357,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Senior Software Engineer to help build core libraries, evaluation systems, and reusable harness capabilities that make AI agents safer, faster, and easier for developers to trust. You will work hands‑on across open‑source agent harnesses, sandboxed execution, model‑provider infrastructure, and agent benchmarks to improve the next generation of agents.

Join our creative engineering team building foundational technology for the future wave of autonomous software systems!

Qualifications

• Bachelor’s or Master’s degree in Computer Science, AI, Applied Math, or equivalent experience.
• 8+ years of hands‑on software‑engineering experience, with production systems ownership.
• Expert Python skills and working proficiency in Rust, Go, C++, or TypeScript.
• Experience building or extending LLM agents, coding agents, tool‑use loops, or evaluation infrastructure.
• Solid understanding of asynchronous execution, subprocesses, containers, sandboxing, callbacks, retries, networking, and distributed compute.
• Experience crafting reproducible experiments, benchmark methodology, performance investigations, regression gates, or reliability analysis under nondeterministic conditions.
• A record of shipping and maintaining open‑source or developer‑facing software with sound testing, API development, documentation, and code reviews.

Responsibilities

• Define and evolve the technical approach for evaluating open agent harnesses across frameworks, models, and benchmark environments.
• Build and operate scalable, reproducible evaluation systems spanning CI, scheduled compute, sandboxed execution, artifact provenance, trace capture, and analysis.
• Diagnose agent failures from traces, logs, tool results, and benchmark artifacts, separating harness, evaluator, environment, inference, and model causes.
• Design controlled experiments and promotion criteria that distinguish genuine harness improvements from noise, benchmark artifacts, or model-specific gains.
• Ship focused harness and runtime improvements, including upstream open‑source contributions, detectors, regression tests, and maintainable documentation.
• Partner with Relay, Hermes, model, infrastructure, and research teams to turn recurring optimization needs into reusable runtime, trace, and evaluation capabilities.

Skills

Python

Rust

Go

C++

TypeScript

Async/Distributed systems

Open‑source contributions

LLM agents / evaluation infra

Education

Bachelor’s or Master’s degree in Computer Science / AI / Applied Math

Tools

OpenTelemetry

OpenInference

AgentBench

CI/CD tooling

Job description

NVIDIA is seeking a Senior Software Engineer to help build core libraries, evaluation systems, and reusable harness capabilities that make AI agents safer, faster, and easier for developers to trust. You will work hands‑on across open‑source agent harnesses, sandboxed execution, model‑provider infrastructure, and agent benchmarks to improve the next generation of agents.

Join our creative engineering team building foundational technology for the future wave of autonomous software systems!