AI Evaluation Engineer for Health Coach Safety

Nuna Inc; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Nuna is building an AI health coach and seeks a data scientist to own the evaluation system end-to-end. You will build the harnesses, datasets, judges, and release gates that ensure safety and quality before any deployment, guiding what ships to patients.

You’ll collaborate with data scientists, clinicians, and engineers while defining standards, methods, and trade-offs in the face of incomplete information.

Responsibilities

• Build testing harnesses and evaluation infrastructure for our agentic products and our internal agentic tooling
• Own our evals end to end - both the architecture and the content - with support from data science and clinical partners
• Make every agentic deployment run through the testing apparatus before it ships, and own the release gates that keep unsafe or low-quality behavior from reaching patients
• Build the ground truth, judges, and metrics, and validate that the evaluation itself can be trusted: calibration to human labels, reliability, and honest confidence on every number, in partnership with our data scientist
• Build functional tooling for labeling and review workflows, so clinicians, coaches, and designers can author and review evaluation scenarios with

Job description

Nuna is building an AI health coach and seeks a data scientist to own the evaluation system end-to-end. You will build the harnesses, datasets, judges, and release gates that ensure safety and quality before any deployment, guiding what ships to patients.

You’ll collaborate with data scientists, clinicians, and engineers while defining standards, methods, and trade-offs in the face of incomplete information.