Software Engineer, Voice Evaluation [NYC or SF]

Concurrence

San Francisco (CA)

On-site

USD 220,000 - 300,000

Full time

2 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
Daily catered meals
Unlimited PTO

Job summary

Concurrence is seeking a Software Engineer, Voice Evaluation to build the system that determines when a voice agent is ready for patients. You will own audio tests, scoring, and the release gate, ensuring failures are caught before release.

You will create simulated callers, test phone trees and interruptions, and build scoring against human labels. You will define release criteria with product, clinical, and engineering teams, and maintain evaluation sets for unseen examples on SF/NYC sites.

Qualifications

  • Candidates have built evaluation or simulation for production agents at scale
  • Experience with voice/audio is preferred
  • Ability to define and justify evaluation score methods

Responsibilities

  • Build simulated callers and replay tests for the production voice path
  • Test phone trees, hold queues, interruptions and escalation
  • Create scoring systems and calibrate judges against human labels
  • Measure failure rates and validate predictions on real calls
  • Define and enforce release criteria with cross-functional teams
  • Turn failures into regression tests and monitor post-release

Skills

Voice evaluation
Simulation systems
Release criteria
Regression tests
Auditory testing

Job description

About Concurrence

Concurrence (formerly Amigo) is building the platform healthcare organizations use to build their own clinical AI agents. Our mission is to make excellent care accessible to every patient, not rationed by the limits of clinician time.

About Concurrence

Concurrence (formerly Amigo) is building the platform healthcare organizations use to build their own clinical AI agents. Our mission is to make excellent care accessible to every patient, not rationed by the limits of clinician time. Underneath our platform is the Health World Model, which unifies a health system’s data into a living, event-level record of every patient’s care. On top of it, agents handle work across the patient journey, from intake and scheduling to medication management and clinical decision support. Our agents have supported more than 10 million patient encounters and are on track to grow tenfold this year. We run natively on Databricks, integrate with every major EHR, and are backed by Madrona, General Catalyst, and Optum Ventures. We’re about 35 people, working in person in New York and San Francisco.

About This Role

As a Software Engineer, Voice Evaluation at Concurrence, you'll build the system that determines whether a voice agent is ready for patients. You'll own realistic audio tests, scoring and the release gate, with authority to block releases that fail it. The tests need to catch failures that a clean transcript can miss, from mishandled interruptions to missed clinical information or a failure to hand off to a person.

What You'll Do
  • Build simulated callers and call-replay tests that exercise the production voice path with real audio
  • Test phone trees, hold queues, interruptions, conflicting clinical information and appropriate escalation
  • Build scoring systems and calibrate automated judges against human assessments
  • Measure clinical failure rates and uncertainty, and test whether simulation results predict behavior on real calls
  • Define release criteria with model, runtime, product and clinical colleagues, own the gate and block changes that fail those criteria
  • Turn production failures into regression tests, verify fixes and monitor behavior after release
  • Maintain held-out evaluation sets so models are tested on examples they did not train against
What We're Looking For
  • You've personally built and owned evaluation or simulation for production agents at scale. Voice and audio experience is a strong preference
  • You've built agent simulations or replay systems, implemented scoring and caught a concrete failure before release
  • Your evaluation results determined what shipped, and you led that work
  • You can explain what a score captures, what it misses and how you checked it against real behavior
  • You write reliable software that runs at volume and follow failures through to a verified fix
  • You use coding agents in your work and independently verify their output
  • You're low ego, direct, and can hold the line on release criteria with evidence
  • You can work on site in New York City or San Francisco
Nice to have
  • Experience calibrating automated judges against human labels
  • Load testing and concurrent calls using audio
  • Speech or audio machine learning
  • Experience in healthcare or another regulated environment
Benefits (available To Full-Time Employees)
Health & Wellness
  • Comprehensive health, dental, and vision insurance
  • Daily catered lunch and dinner
  • Mental health support and wellness coaching
  • Flexible wellness stipend for fitness, therapy, or personal growth
Growth & Development
  • Annual learning budget for courses, books, or conferences
  • Conference attendance budget for professional development
  • Annual team offsite
  • Academic collaboration opportunities
  • Unlimited PTO
Our Core Values
  • Patients Win, We Win

If patients aren't getting better care, we haven't earned the right to scale. Every internal decision gets pressure-tested: does this make patients' lives better? If we can't draw the line, we question why we're doing it.

  • High Standards, High Care
  • Thoughtful Urgency
  • Intensely Measured

We instrument patient outcomes, provider ROI, system performance, and clinical accuracy. But data without action is surveillance. Every metric should have an owner, a threshold, and a response plan. If we're measuring something but never acting on it, we stop measuring it.

Who Builds With Us
  • Low ego: Politics and territory don't interest you. The best ideas win, regardless of who has them.
  • Direct: You say the hard thing, challenge ideas openly, and commit fully once decided.
  • High agency: You thrive on trust rather than instruction. When you see something is broken, you fix it. You don’t file tickets and wait for someone else.
  • Bar of excellence: You hold yourself to a bar most people wouldn't, and you want teammates who do the same.
  • Skeptical: You push back on rules that don’t make sense and question assumptions that haven’t earned their place.

Compensation Range: $220K - $300K

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
Software Engineer, Voice Evaluation [NYC or SF]
Software Engineer, Voice Evaluation [NYC or SF]

Concurrence Inc • New York (NY)

On-site
USD 120,000 - 170,000
Health insurance
Daily catered meals
Mental health support
+4
Research Engineer, Voice AI [NYC or SF]
Research Engineer, Voice AI [NYC or SF]

Concurrence • New York (NY)

On-site
USD 220,000 - 300,000
Health benefits
Lunch provided
Wellness stipend
+2
Research Engineer, Voice AI [NYC or SF]
Research Engineer, Voice AI [NYC or SF]

Concurrence • San Francisco (CA)

On-site
USD 220,000 - 300,000
Health insurance
Daily catered lunch
Mental health support and wellness
+4
Senior Software Engineer - Backend/Infra [NYC or SF]
Senior Software Engineer - Backend/Infra [NYC or SF]

Concurrence • New York (NY)

On-site
USD 220,000 - 260,000
Health insurance
Daily catered lunch and dinner
Mental health support and wellness
+4
Senior Applied AI Engineer (Forward Deployed) [NYC or SF]
Senior Applied AI Engineer (Forward Deployed) [NYC or SF]

Concurrence • New York (NY)

On-site
USD 200,000 - 260,000
Health insurance
Lunch and dinner provided
Wellness stipend
+1
Future Opportunities [NYC or SF]
Future Opportunities [NYC or SF]

Concurrence • San Francisco (CA)

On-site
USD 150,000 - 210,000
Comprehensive health, dental, and vis
Daily catered lunch and dinner
Mental health support and wellness
+6
Staff Software Engineer - Full Stack [NYC or SF]
Staff Software Engineer - Full Stack [NYC or SF]

Concurrence • San Francisco (CA)

On-site
USD 180,000 - 240,000
Health insurance
Daily catered meals
Mental health support
+2
Staff Applied AI Engineer (Forward Deployed) [NYC or SF]
Staff Applied AI Engineer (Forward Deployed) [NYC or SF]

Concurrence • New York (NY)

On-site
USD 240,000 - 300,000
Health insurance
Lunch provided
Mental health support
+4
Senior Software Engineer - Backend/Infra [NYC or SF]
Senior Software Engineer - Backend/Infra [NYC or SF]

Concurrence Inc • New York (NY)

On-site
USD 180,000 - 240,000
Health insurance
Catered meals
Mental health support
+6