Software Engineer (AI Quality & Reliability)

Taxo; New York (NY); On-site; USD 130,000 - 170,000; Full time

Benefits offered by this job

Health coverage

Job summary

Taxo is hiring a Software Engineer (AI Quality & Reliability) to build automated evaluation and regression systems for our Voice AI agents. You will turn product criteria and real failure modes into repeatable tests across conversations, tool calls, and integrations, ensuring reliable releases.

You will own evaluation frameworks, measure AI behavior, and surface regressions before launch, collaborating with engineering and product teams in an on-site New York setting.

Qualifications

• Strong software engineering skills in Python or TypeScript.
• Experience building test automation and CI pipelines.
• Experience evaluating LLM-powered apps or voice systems.
• Ability to trace failures across service boundaries.
• Ability to communicate risk and release readiness.

Responsibilities

• Build evaluation system and test harnesses.
• Automate end-to-end regression across workflows.
• Measure AI behavior including latency and scoring.
• Produce release-readiness evidence and dashboards.
• Turn failures into durable regression tests.
• Improve test infrastructure and CI/CD integration.

Skills

Python

TypeScript

Test automation

CI/CD

Distributed systems

LLM evaluation

Healthcare data

Job description

Software Engineer (AI Quality & Reliability)

Location

New York City

Employment Type

Full time

Location Type

On-site

Department

Engineering

The Mission

Every day, patients call their clinic to schedule an appointment, ask a question, or request a prescription refill. Behind each conversation is a busy healthcare team coordinating people, information, and systems.

Taxo builds Voice AI agents that take on this administrative work. Our agents connect with EHR and practice management systems to help teams handle patient intake, complex scheduling, and refill requests, with reliable workflows and clear handoffs to people.

The Opportunity: Make AI Quality Measurable

As our Software Engineer (AI Quality & Reliability), you'll build the automated evaluation and regression systems that show whether Taxo is ready to release. You'll turn product acceptance criteria and real failure modes into repeatable tests across voice agents, application workflows, and integrations. This is a software engineering role focused on test infrastructure, independent quality evidence, and reliable releases.

What You'll Do

• Build the Evaluation System: Own shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or appropriately authorized, privacy‑safe data. Make runs repeatable and results easy to investigate.
• Test the Whole Workflow: Build automated end‑to‑end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation.
• Measure AI Behavior: Evaluate task completion, tool‑use correctness, latency, and conversation quality. Combine deterministic checks with calibrated model‑assisted scoring and appropriate human review.
• Make Releases Reviewable: Produce release‑readiness evidence, maintain quality dashboards, and coordinate quality gates with product and engineering. Surface regressions, coverage gaps, and unresolved risks before launch.
• Turn Failures into Tests: Reproduce defects, isolate failure patterns, and add durable regression cases. Engineering service owners retain component tests, production instrumentation, on‑call response, and fixes; you independently verify the results.
• Improve the Test System: Integrate automated checks into CI/CD, reduce flaky tests, and keep test environments and fixtures useful as the product evolves. Partner with domain reviewers on healthcare‑specific cases and elevate questions that require clinical judgment.

Who You Are

• The Test Engineer: Strong software engineering skills in Python or TypeScript and experience building maintainable test automation, reusable fixtures, and CI pipelines.
• The AI Evaluator: Practical experience evaluating LLM‑powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring.
• The Systems Thinker: Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries.
• The Investigator: A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage.
• The Quality Partner: Clear communication about risk and release readiness, with sound judgment around sensitive data and human review.
• Bonus: Experience with streaming audio, telephony, healthcare workflows, EHR integrations, or reliability testing in privacy‑sensitive products.

Why Taxo

Healthcare teams should be able to spend more time on patients and less time on administration. Help us build the products and company that make that possible.

• Practical AI Work: Build evaluation systems for real‑world AI behavior across healthcare workflows.
• Impact That Matters: Help catch failures before they disrupt patients or clinic teams.
• Technical Ownership: Shape shared evaluation and test infrastructure that improves every release.

Compensation

Base salary: $130,000–$170,000 USD annually

Equity: 0.10%–0.25%

Benefits: Health coverage

How We Work

Taxo is a small, fast, high‑conviction team. We care deeply about the problem, move with urgency, and take ownership of the work. We collaborate closely, make practical decisions, and keep improving through what we learn from customers.

If you want to build the engineering systems that make healthcare AI more dependable, we'd love to hear from you.