Voice Evaluation Engineer — AI Medical Agents

Concurrence

New York (NY)

On-site

USD 220,000 - 300,000

Full time

3 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Concurrence is hiring a Software Engineer, Voice Evaluation to build the system that judges whether a voice agent is ready for patients. You will own audio tests, scoring, and the release gate, ensuring failures are caught before patient interaction.

You will create simulated callers, calibrate automated judges against human assessments, and measure failure rates to improve model reliability. On-site in New York City or San Francisco, with a strong emphasis on scalable, fault-tolerant software.

Qualifications

  • You've personally built and owned evaluation or simulation for production agents at scale. Voice and audio experience is a strong preference
  • You've built agent simulations or replay systems, implemented scoring and caught a concrete failure before release
  • Your evaluation results determined what shipped, and you led that work
  • You can explain what a score captures, what it misses and how you checked it against real behavior
  • You write reliable software that runs at volume and follow failures through to a verified fix
  • You use coding agents in your work and independently verify their output
  • You're low ego, direct, and can hold the line on release criteria with evidence
  • You can work on site in New York City or San Francisco

Responsibilities

  • Build simulated callers and call-replay tests that exercise the production voice path with real audio
  • Test phone trees, hold queues, interruptions, conflicting clinical information and appropriate escalation
  • Build scoring systems and calibrate automated judges against human assessments
  • Measure clinical failure rates and uncertainty, and test whether simulation results predict behavior on real calls
  • Define release criteria with model, runtime, product and clinical colleagues, own the gate and block changes that fail those criteria
  • Turn production failures into regression tests, verify fixes and monitor behavior after release
  • Maintain held-out evaluation sets so models are tested on examples they did not train against

Job description

Concurrence is hiring a Software Engineer, Voice Evaluation to build the system that judges whether a voice agent is ready for patients. You will own audio tests, scoring, and the release gate, ensuring failures are caught before patient interaction.

You will create simulated callers, calibrate automated judges against human assessments, and measure failure rates to improve model reliability. On-site in New York City or San Francisco, with a strong emphasis on scalable, fault-tolerant software.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Voice Evaluation Engineer for Scalable Clinical AI
Voice Evaluation Engineer for Scalable Clinical AI

Concurrence Inc • New York (NY)

On-site
USD 120,000 - 170,000
Health insurance
Daily catered meals
Mental health support
+4
Voice AI Evaluation Engineer - Release Gate Lead
Voice AI Evaluation Engineer - Release Gate Lead

Concurrence • San Francisco (CA)

On-site
USD 220,000 - 300,000
Health insurance
Daily catered meals
Unlimited PTO
Voice AI Research Engineer, Real-Time Clinical Speech
Voice AI Research Engineer, Real-Time Clinical Speech

Concurrence • San Francisco (CA)

On-site
USD 220,000 - 300,000
Health insurance
Daily catered lunch
Mental health support and wellness
+4
Voice AI Evaluation Engineer
Voice AI Evaluation Engineer

Aircall • San Francisco (CA)

On-site
USD 181,000 - 250,000
Competitive salary package & benefits
Voice AI Research Engineer — Real-Time Clinical Speech
Voice AI Research Engineer — Real-Time Clinical Speech

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
Health benefits
Lunch provided
Wellness stipend
+2
Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence • San Francisco (CA)

On-site
USD 220,000 - 300,000
Health insurance
Daily catered meals
Unlimited PTO
Senior ML Engineer — AI Agent Evaluation & Voice Models
Senior ML Engineer — AI Agent Evaluation & Voice Models

Jobot • Austin (TX)

On-site
USD 200,000 - 300,000
Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
ML Engineer: Voice Models & Evaluation Architect
ML Engineer: Voice Models & Evaluation Architect

Aircall.io, Inc. • San Francisco (CA)

On-site
USD 181,000 - 250,000
Competitive salary package
Growth opportunities
Remote Voice QA & AI Training Evaluator
Remote Voice QA & AI Training Evaluator

Inworld AI • United States

Remote
USD 28,000 - 76,000