AI Quality & Reliability Engineer - NYC Healthcare

Taxo.ai, Inc.

New York (NY)

On-site

USD 130,000 - 170,000

Full time

4 days ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health coverage

Job summary

Taxo.ai, Inc. is seeking a Software Engineer focused on AI Quality & Reliability in New York City. You will build automated evaluation and regression systems for voice AI agents, ensuring reliable releases and measurable quality across conversations, tool calls, and integrations.

The role emphasizes test infrastructure, reproducible evidence, and coordination with product and engineering to surface regressions before launch. In-office in NYC with competitive compensation.

Qualifications

Experience building maintainable test automation and reusable fixtures.
Experience evaluating LLM-powered applications, agents, or voice systems.
Experience testing APIs, asynchronous workflows, and distributed systems.
Ability to turn ambiguous failures into reproducible test cases.

Responsibilities

Build the evaluation system and data pipelines for repeatable tests.
Create end-to-end regression coverage across conversations and integrations.
Measure AI behavior, latency, and conversation quality with automated checks.
Produce release-readiness evidence and dashboards for quality gates.
Reproduce defects and add durable regression cases.
Improve the test system and CI/CD integration with healthcare review partners.

Skills

Test automation

Python/TypeScript

CI pipelines

API testing

LLM evaluation

Tools

CI/CD tooling

API testing tools

Job description

The role emphasizes test infrastructure, reproducible evidence, and coordination with product and engineering to surface regressions before launch. In-office in NYC with competitive compensation.

Get your free, confidential resume review.

or drag and drop your file here.