Software Engineer, AI Evaluation

nuna

United States

Presencial

USD 140.000 - 180.000

Tempo integral

14 dias+

Uma candidatura completa em um minuto

Currículo e carta de apresentação personalizados, prontos para serem enviados.

Passe pelos filtros ATS

Resumo da vaga

Nuna is building an AI health coach and seeks a role that owns how we evaluate our agents end to end. You will design the evaluation harnesses, datasets, judges, and release gates ensuring the coach is safe and high quality before shipping.

You will collaborate with clinicians, data scientists, and engineers to define standards and trade-offs in a fast-moving environment. This is a hands-on role focused on building production-grade evaluation systems, calibrating to human labels, and enabling

Qualificações

  • Significant experience shipping reliable production systems and tooling.
  • Deep understanding of evaluating AI systems and how evals fail.
  • Testing mindset for building measurement systems; regression discipline.

Responsabilidades

  • Build testing harnesses and evaluation infrastructure for agentic products and internal tooling.
  • Own end-to-end evals—the architecture and content—with data science and clinical partners.
  • Ensure every agentic deployment passes through the testing apparatus and release gates.
  • Create ground truth, judges, and metrics; calibrate to human labels and ensure reliability.
  • Build labeling and review tooling so clinicians, coaches, and designers can author evaluation scenarios.
  • Iterate evaluation results into model and prompt refinement with reduced human hand-holding.

Conhecimentos

Production systems
AI evaluation methods
Multi-turn evaluation
Data labeling workflows
UI for clinicians

Ferramentas

LangSmith
Braintrust
DeepEval
Ragas
Promptfoo

Descrição da vaga

Chronic disease isn't managed in a clinic. It is managed at home, in relationships, in the everyday. What's on the dinner table, what gets said, and who notices when someone's struggling. For the 130 million Americans managing a chronic condition, the healthcare system has offered the same answer for decades: a 15-minute doctor's visit, a pamphlet, and a portal login they'll never use.

At Nuna, we are building an AI health coach that shows up like a person who actually has time: available at 3am, infinitely patient, and never behind a waiting room. We use motivational interviewing to help patients and their families see themselves clearly, design experiments that fit their real lives, and navigate a system that has not historically been on their side. We are building from the ground up around a simple belief: patients don't want to be healthy; they want their lives back.

We're not competing with other health apps. We're competing with the moment a person gives up on getting better. If that's a problem you want to work on, we'd like to talk.

Your team

We are a small, interdisciplinary team - engineers, data scientists, designers, product managers, and clinicians - building Nuna's AI health coach. Our products are only as good as the care and science behind them, and your piece is how we know the coach is safe and working. You'll own the evaluation system for the team building the coach: a data scientist partners with you on the science, clinicians and designers supply the ground truth, and the engineers shipping the agents depend on the signal you produce to decide what ships.

The role

This is a net-new, build-first role for someone who wants to own how we evaluate our AI agents end to end. You'll build the harnesses, datasets, judges, and release gates that tell us whether the coach is safe and good, and you'll own both that infrastructure and the evals that run on it. This is not a test-execution role - you write the code and own the system, rather than running tests someone else designed. You'll make the day-to-day calls on standards, methods, and trade-offs, often with incomplete information and the freedom to define the right answer yourself. Comfort with ambiguity is part of the job.

What you'll do
  • Build testing harnesses and evaluation infrastructure for our agentic products and our internal agentic tooling
  • Own our evals end to end - both the architecture and the content - with support from data science and clinical partners
  • Make every agentic deployment run through the testing apparatus before it ships, and own the release gates that keep unsafe or low-quality behavior from reaching patients
  • Build the ground truth, judges, and metrics, and validate that the evaluation itself can be trusted: calibration to human labels, reliability, and honest confidence on every number, in partnership with our data scientist
  • Build functional tooling for labeling and review workflows, so clinicians, coaches, and designers can author and review evaluation scenarios without an engineer in the loop
  • Help close the loop from evaluation results to model and prompt refinement, working toward systems that iterate safely with less human hand-holding
What we're looking for
  • Significant experience building and shipping reliable production systems and tooling
  • Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You've deployed evals and automated AI tooling in production, not just prototyped them
  • A testing mindset applied to building the measurement system, not running tests against a spec: adversarial instinct, coverage thinking, regression discipline, and documentation others can build on
  • You use AI in your daily work and build tools that make the people around you more effective
  • Enough fluency in statistics and experimental design to partner with a data scientist on calibration and reliability
  • Can design the workflow and build a functional UI for non-engineers like clinicians and labelers
  • A genuine interest in improving healthcare alongside an interdisciplinary team, with the judgment to tell a launch-blocking issue from a nice-to-have
Bonus Points
  • Experience in healthcare or another regulated, high-trust domain, and familiarity with the regulatory landscape
  • Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)
  • Red-teaming or AI safety experience - prompt injection, jailbreaks, adversarial and stress testing
  • Experience with automated, eval-driven model or prompt optimization
  • You've built in an early-stage or fast-moving environment

Nuna is an Equal Employment Opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, genetics and/or veteran status.

Obtenha sua revisão de currículo grátis e confidencial.

ou arraste e solte seu arquivo aqui.

Similar jobs

Vagas similares que valem a pena comparar

Software Engineer, AI Evaluation
Software Engineer, AI Evaluation

Nuna • San Francisco (CA)

Presencial
USD 150.000 - 230.000
Senior Software Engineer
Senior Software Engineer

Nuna • San Francisco (CA)

Presencial
USD 120.000 - 150.000
Software Engineer
Software Engineer

Nuna • San Francisco (CA)

Híbrido
USD 150.000 - 190.000
Software Engineer
Software Engineer

Nuna Inc. • San Francisco (CA)

Presencial
USD 140.000 - 190.000
Senior Software Engineer
Senior Software Engineer

nuna • EUA

Presencial
USD 140.000 - 190.000
Health insurance
Life insurance
Retirement benefits
+2
Research Engineer, Evals & RL Environments
Research Engineer, Evals & RL Environments

Nolla Health • New York (NY)

Presencial
USD 180.000 - 240.000
Health Insurance
HSA/FSA Eligible
Flexible, unlimited vacation
+2
AI Evaluation Engineer
AI Evaluation Engineer

Bask Health • New York (NY)

Presencial
USD 120.000 - 170.000
Founding AI Engineer, Agents & Evaluation
Founding AI Engineer, Agents & Evaluation

Zingage • New York (NY)

Presencial
USD 190.000 - 240.000
Competitive base and meaningful equity
Equipment stipend
Luxury gym membership in NYC
+4
Senior AI Engineer – Agents
Senior AI Engineer – Agents

MiraMace, Inc. • San Francisco (CA)

Presencial
USD 150.000 - 210.000
AI Product Engineer
AI Product Engineer

3Y • EUA

Remoto
USD 150.000 - 190.000