Human Evaluation Researcher

Nuance Labs; Seattle (WA); On-site; USD 160,000 - 190,000; Full time

Benefits offered by this job

Visa sponsorship

Health plans

Commuter benefits

401(k) match

PTO and holidays

Free lunch and snacks

Job summary

Nuance Labs in Seattle is seeking an experienced human-evaluation scientist to turn subjective judgments into measurable signals for real-time AI avatars. You will design studies, define rubrics, and coordinate with researchers to align outcomes with product goals.

You will own the end-to-end evaluation pipeline, from study design to data analysis, and you’ll collaborate directly with founders and the modeling team to decide which models ship next.

Qualifications

• 5+ years designing and running human-subjects research in industry or academia.
• Examples of studies leading to converged ambiguous judgments with rubrics, anchors and protocols.
• Strong grounding in qualitative and quantitative methods.
• Familiarity with agreement metrics (e.g., Cohen's kappa, Krippendorff's alpha).
• Ability to explain findings crisply to ML researchers and engineers.

Responsibilities

• Design and run qualitative and quantitative studies of our AI avatars: side-by-side comparisons, controlled rating experiments, interviews, think-alouds, diary studies and panels.
• Turn ambiguous judgments into reliable instruments (rubrics, anchors, protocols) and measure inter-rater agreement without dulling signal.
• Use ethnographic techniques (observation, contextual inquiry) to understand real user experience of face-to-face AI.
• Build the human-eval pipeline itself: participant panels, rater training and calibration, tooling and evaluation cadence tied to model releases.
• Calibrate automated and model-based metrics against human judgment to guide trust and release decisions.

Skills

Human-subjects research design

Qualitative & quantitative methods

Inter-rater reliability

Study design and rubrics

Communication with ML researchers

Education

MS/PhD in related field

Job description

Nuance Labs is building photorealistic, real-time AI avatars with emotional intelligence: a full-duplex audiovisual system that can listen, speak, react, interrupt, and respond like a real person.

We're a research company, with PhDs from MIT, UW, Oxford, CMU, and Johns Hopkins, and industry experience from Apple, Meta, Amazon AGI, and more. Backed by Accel, Lightspeed, South Park Commons, and NVIDIA, we combine frontier research with ruthless engineering needed for consumer-grade, real-time systems. The team is small, the work is real, and the problems are unsolved.

How Nuance Differentiates

Most conversational AI avatars today are hacks — a face slapped on a speech-to-speech pipeline, stuck in the uncanny valley: emotionless, mechanical, one-turn-at-a-time. Current systems take 2-5 seconds to respond; natural conversation requires sub-500ms. That's a 10x improvement, and it demands rethinking the entire stack.

That rethinking starts with full-duplex: an AI that listens and speaks simultaneously, perceives emotion in real time, and responds with a face that actually reflects it. It's an extremely hard problem, and we're developing foundation models designed for it from the ground up.

Why this role exists

"Does this avatar feel human?" is the question our whole company is organized around — and no automated metric can answer it. Lip-sync error and video quality scores say nothing about whether a smile landed as sincere or unsettling, whether a conversation felt warm or hollow, or whether someone would want to talk to our avatar again tomorrow.

Your job is to turn human judgment into a reliable signal our researchers can train and ship against. You'll take the most ambiguous problems in our field (naturalness, emotional resonance, trust, presence) and design studies whose results people actually agree on. When two models differ, your study is the tiebreaker. When a model "feels off" and nobody can say why, your investigation finds the cause.

This is a hands-on IC role and our first hire dedicated to human evaluation. You'll own it end to end (what to measure, how to measure it, who rates it, and what the results mean) working directly with the founders and the modeling team. Your findings will decide which models ship and what we train next.

What you'll do

• Design and run qualitative and quantitative studies of our AI avatars: side-by-side comparisons, controlled rating experiments, in-depth interviews, think-alouds, diary studies, and longitudinal panels.
• Turn ambiguous judgments into instruments people can align on (rubrics, anchored scales, annotation guidelines) then measure and improve inter-rater agreement without flattening real signal.
• Use ethnographic techniques (observation of live conversations, contextual inquiry, field work) to understand how people actually experience face-to-face AI, not just what they report in a survey.
• Build the human-eval pipeline itself: participant panels, rater training and calibration, tooling, and an evaluation cadence tied to model releases.
• Calibrate automated and model-based metrics (including LLM-as-judge) against human judgment, so the team knows when to trust them and when not to.

You may be a good fit if you have

• 5+ years designing and running human-subjects research in industry or academia (UX research, HCI, experimental psychology, behavioral science, or a related field).
• Examples of study designs you can walk us through, especially ones where you got humans to converge on an ambiguous judgment (tone, emotion, quality, trust), including the rubrics, anchors, and protocols that made alignment possible.
• Strong grounding in both qualitative methods (interviews, ethnography, contextual inquiry) and quantitative methods (survey and psychometric design, experimental design, statistics for rating and pairwise-comparison data).
• Fluency with agreement and reliability. For example: You know your Cohen's kappa from your Krippendorff's alpha, and more importantly, how to raise them.
• A bias toward running a scrappy, sound study this week over a perfect one next quarter, and the ability to explain findings crisply to ML researchers.

Strong candidates may also have

• Experience evaluating generative AI: avatars or digital humans, speech or video generation, conversational agents, or emotion expression and recognition.
• A background in perceptual science or psychophysics (how people perceive faces, voices, motion, and emotion). MS/PhD in a related field welcome.
• Familiarity with human evaluation at scale: crowdsourcing platforms, annotation tooling, golden datasets.
• Enough statistics and scripting (Python or R) to analyze your own data.

No candidate checks every box. If the "good fit" list sounds like you but your background is unconventional, we'd like to hear from you anyway.

Compensation

$160,000 - $190,000 base salary, plus meaningful equity. We think long-term ownership matters and structure equity accordingly.

Logistics

• Location:In-person in Seattle, five days a week - we believe in the compounding value of working shoulder-to-shoulder.
• Visa sponsorship:We sponsor visas (O-1, H-1B, green card, etc.) from day one.
• AI-native tooling:Do your best work with the best tools, including unlimited tokens.
• Health: We offer a variety of plans that meet your needs, including an HDHP with ~$2,000 in annual HSA contributions by the company (roughly 2x what most big tech companies put in).
• Time off: 15 days of PTO, 10 public holidays, and we close the office for a full week at year-end.
• Food: Lunch, drinks, and snacks on us every workday. We observe boba Tuesdays and Thursdays.
• Commuter benefits:Utilize pre-tax money (up to $340/month) for parking and transportation.
• 401(k):4% match (100% of 1st 1% + 50% of next 5% contributions).