AI Evaluation Engineer — Build Safe, Deploy-Ready Agents

nuna

United States

On-site

USD 140,000 - 180,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Nuna is building an AI health coach and seeks a role that owns how we evaluate our agents end to end. You will design the evaluation harnesses, datasets, judges, and release gates ensuring the coach is safe and high quality before shipping.

You will collaborate with clinicians, data scientists, and engineers to define standards and trade-offs in a fast-moving environment. This is a hands-on role focused on building production-grade evaluation systems, calibrating to human labels, and enabling

Qualifications

  • Significant experience shipping reliable production systems and tooling.
  • Deep understanding of evaluating AI systems and how evals fail.
  • Testing mindset for building measurement systems; regression discipline.

Responsibilities

  • Build testing harnesses and evaluation infrastructure for agentic products and internal tooling.
  • Own end-to-end evals—the architecture and content—with data science and clinical partners.
  • Ensure every agentic deployment passes through the testing apparatus and release gates.
  • Create ground truth, judges, and metrics; calibrate to human labels and ensure reliability.
  • Build labeling and review tooling so clinicians, coaches, and designers can author evaluation scenarios.
  • Iterate evaluation results into model and prompt refinement with reduced human hand-holding.

Skills

Production systems
AI evaluation methods
Multi-turn evaluation
Data labeling workflows
UI for clinicians

Tools

LangSmith
Braintrust
DeepEval
Ragas
Promptfoo

Job description

Nuna is building an AI health coach and seeks a role that owns how we evaluate our agents end to end. You will design the evaluation harnesses, datasets, judges, and release gates ensuring the coach is safe and high quality before shipping.

You will collaborate with clinicians, data scientists, and engineers to define standards and trade-offs in a fast-moving environment. This is a hands-on role focused on building production-grade evaluation systems, calibrating to human labels, and enabling

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Evaluation & Safety Engineer
AI Evaluation & Safety Engineer

Nuna • San Francisco (CA)

On-site
USD 150,000 - 230,000
Software Engineer, AI Evaluation
Software Engineer, AI Evaluation

nuna • United States

On-site
USD 140,000 - 180,000
Software Engineer, AI Evaluation
Software Engineer, AI Evaluation

Nuna • San Francisco (CA)

On-site
USD 150,000 - 230,000
Lead AI Evaluation Engineer — Agent Quality
Lead AI Evaluation Engineer — Agent Quality

FurtherAI • San Francisco (CA)

On-site
USD 140,000 - 210,000
Fully covered health, dental, and visa
Competitive compensation, stock + 401k
Unlimited PTO
+4
Senior AI Agent Evaluation Engineer
Senior AI Agent Evaluation Engineer

NVIDIA Corporation • Washington

On-site
USD 184,000 - 357,000
AI Evaluation & Safety Engineer
AI Evaluation & Safety Engineer

Syntic AI, Inc. • United States

Remote
USD 120,000 - 160,000
AI Evaluation Architect
AI Evaluation Architect

Nouvells • United States

Remote
USD 90,000 - 130,000
Senior AI Systems Engineer - Evaluation & Reliability
Senior AI Systems Engineer - Evaluation & Reliability

ExpertsHub.ai • San Francisco (CA)

On-site
USD 250,000 - 360,000
Equity
Medical Insurance
Dental Insurance
+4
Senior AI Eval & Agent-Quality Engineer
Senior AI Eval & Agent-Quality Engineer

Town.com • San Francisco (CA)

On-site
USD 180,000 - 250,000
AI Automation Engineer - End-to-End Evaluation
AI Automation Engineer - End-to-End Evaluation

Niantic • San Francisco (CA)

Hybrid
USD 158,000 - 210,000