Voice AI Evaluation Engineer - Release Gate Lead

Concurrence

San Francisco (CA)

On-site

USD 220,000 - 300,000

Full time

2 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Daily catered meals
Unlimited PTO

Job summary

Concurrence is seeking a Software Engineer, Voice Evaluation to build the system that determines when a voice agent is ready for patients. You will own audio tests, scoring, and the release gate, ensuring failures are caught before release.

You will create simulated callers, test phone trees and interruptions, and build scoring against human labels. You will define release criteria with product, clinical, and engineering teams, and maintain evaluation sets for unseen examples on SF/NYC sites.

Qualifications

  • Candidates have built evaluation or simulation for production agents at scale
  • Experience with voice/audio is preferred
  • Ability to define and justify evaluation score methods

Responsibilities

  • Build simulated callers and replay tests for the production voice path
  • Test phone trees, hold queues, interruptions and escalation
  • Create scoring systems and calibrate judges against human labels
  • Measure failure rates and validate predictions on real calls
  • Define and enforce release criteria with cross-functional teams
  • Turn failures into regression tests and monitor post-release

Skills

Voice evaluation
Simulation systems
Release criteria
Regression tests
Auditory testing

Job description

Concurrence is seeking a Software Engineer, Voice Evaluation to build the system that determines when a voice agent is ready for patients. You will own audio tests, scoring, and the release gate, ensuring failures are caught before release.

You will create simulated callers, test phone trees and interruptions, and build scoring against human labels. You will define release criteria with product, clinical, and engineering teams, and maintain evaluation sets for unseen examples on SF/NYC sites.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Voice Evaluation Engineer — AI Medical Agents
Voice Evaluation Engineer — AI Medical Agents

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
Voice Evaluation Engineer for Scalable Clinical AI
Voice Evaluation Engineer for Scalable Clinical AI

Concurrence Inc • New York (NY)

On-site
USD 120,000 - 170,000
Health insurance
Daily catered meals
Mental health support
+4
Voice AI Evaluation Engineer
Voice AI Evaluation Engineer

Aircall • San Francisco (CA)

On-site
USD 181,000 - 250,000
Competitive salary package & benefits
Senior ML Engineer — AI Agent Evaluation & Voice Models
Senior ML Engineer — AI Agent Evaluation & Voice Models

Jobot • Austin (TX)

On-site
USD 200,000 - 300,000
Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence • San Francisco (CA)

On-site
USD 220,000 - 300,000
Health insurance
Daily catered meals
Unlimited PTO
Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
ML Engineer: Voice Models & Evaluation Architect
ML Engineer: Voice Models & Evaluation Architect

Aircall.io, Inc. • San Francisco (CA)

On-site
USD 181,000 - 250,000
Competitive salary package
Growth opportunities
Founding AI Voice Systems Engineer (Real-Time Eval)
Founding AI Voice Systems Engineer (Real-Time Eval)

Roark • United States

Remote
USD 150,000 - 190,000
Senior QA Engineer, Voice & LLM for Healthcare AI
Senior QA Engineer, Voice & LLM for Healthcare AI

Vi • United States

Remote
USD 120,000 - 180,000
Founding Software Engineer — AI & Voice Systems
Founding Software Engineer — AI & Voice Systems

Roark • United States

Remote
USD 150,000 - 190,000