Software Engineer (AI Quality & Reliability)

Taxo.ai, Inc.; New York (NY); On-site; USD 130,000 - 170,000; Full time

Benefits offered by this job

Health coverage

Job summary

Taxo.ai, Inc. is seeking a Software Engineer focused on AI Quality & Reliability in New York City. You will build automated evaluation and regression systems for voice AI agents, ensuring reliable releases and measurable quality across conversations, tool calls, and integrations.

The role emphasizes test infrastructure, reproducible evidence, and coordination with product and engineering to surface regressions before launch. In-office in NYC with competitive compensation.

Qualifications

• Experience building maintainable test automation and reusable fixtures.
• Experience evaluating LLM-powered applications, agents, or voice systems.
• Experience testing APIs, asynchronous workflows, and distributed systems.
• Ability to turn ambiguous failures into reproducible test cases.

Responsibilities

• Build the evaluation system and data pipelines for repeatable tests.
• Create end-to-end regression coverage across conversations and integrations.
• Measure AI behavior, latency, and conversation quality with automated checks.
• Produce release-readiness evidence and dashboards for quality gates.
• Reproduce defects and add durable regression cases.
• Improve the test system and CI/CD integration with healthcare review partners.

Skills

Test automation

Python/TypeScript

CI pipelines

API testing

LLM evaluation

Tools

CI/CD tooling

API testing tools

Job description

Software Engineer (AI Quality & Reliability)

Location: New York City | In-office

The Mission

“Leading healthcare into the agentic era.”

Every day, patients call their clinic to schedule an appointment, ask a question, or request a prescription refill. Behind each conversation is a busy healthcare team coordinating people, information, and systems.

Taxo builds Voice AI agents that take on this administrative work. Our agents connect with EHR and practice management systems to help teams handle patient intake, complex scheduling, and refill requests, with reliable workflows and clear handoffs to people.

The Opportunity: Make AI Quality Measurable

As our Software Engineer (AI Quality & Reliability), you’ll build the automated evaluation and regression systems that show whether Taxo is ready to release. You’ll turn product acceptance criteria and real failure modes into repeatable tests across voice agents, application workflows, and integrations. This is a software engineering role focused on test infrastructure, independent quality evidence, and reliable releases.

What You’ll Do

• Build the Evaluation System: Own shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or appropriately authorized, privacy-safe data. Make runs repeatable and results easy to investigate.

• Test the Whole Workflow: Build automated end-to-end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation.

• Measure AI Behavior: Evaluate task completion, tool-use correctness, latency, and conversation quality. Combine deterministic checks with calibrated model-assisted scoring and appropriate human review.

• Make Releases Reviewable: Produce release-readiness evidence, maintain quality dashboards, and coordinate quality gates with product and engineering. Surface regressions, coverage gaps, and unresolved risks before launch.

• Turn Failures into Tests: Reproduce defects, isolate failure patterns, and add durable regression cases. Engineering service owners retain component tests, production instrumentation, on-call response, and fixes; you independently verify the results.

• Improve the Test System: Integrate automated checks into CI/CD, reduce flaky tests, and keep test environments and fixtures useful as the product evolves. Partner with domain reviewers on healthcare-specific cases and elevate questions that require clinical judgment.

Who You Are

• The Test Engineer: Strong software engineering skills in Python or TypeScript and experience building maintainable test automation, reusable fixtures, and CI pipelines.

• The AI Evaluator: Practical experience evaluating LLM-powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring.

• The Systems Thinker: Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries.

• The Investigator: A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage.

• The Quality Partner: Clear communication about risk and release readiness, with sound judgment around sensitive data and human review.

• Bonus: Experience with streaming audio, telephony, healthcare workflows, EHR integrations, or reliability testing in privacy-sensitive products.

Why Taxo

Healthcare teams should be able to spend more time on patients and less time on administration. Help us build the products and company that make that possible.

• Practical AI Work: Build evaluation systems for real-world AI behavior across healthcare workflows.

• Impact That Matters: Help catch failures before they disrupt patients or clinic teams.

• Technical Ownership: Shape shared evaluation and test infrastructure that improves every release.

Compensation

Base salary: $130,000–$170,000 USD annually

Equity: 0.10%–0.25%

Benefits: Health coverage

How We Work

Taxo is a small, fast, high-conviction team. We care deeply about the problem, move with urgency, and take ownership of the work. We collaborate closely, make practical decisions, and keep improving through what we learn from customers.

If you want to build the engineering systems that make healthcare AI more dependable, we’d love to hear from you.