Senior AI QA & Evaluation Engineer

Third Way Health; Cambridge (MA); On-site; USD 130,000 - 180,000; Full time

Job summary

Third Way Health is seeking a Senior QA / Eval Engineer to own and evolve the quality and evaluation infrastructure behind our AI-powered patient engagement platform. You will shape how we define quality across automated and manual workflows and build tooling to make that definition measurable and actionable.

You will collaborate with ML engineers, product managers, and operations leads to translate real-world failure modes into automated checks and contribute to the QA pipeline orchestration

Qualifications

• 5+ years of software engineering or test engineering experience, with 3+ years focused on quality infrastructure for AI/ML or data-intensive systems.
• Strong proficiency in Python, particularly for building test frameworks, eval pipelines, and API-level integration tests (e.g., pytest, FastAPI TestClient, Pydantic).
• Demonstrated experience designing evaluation or verification systems for LLM-based applications — with a clear understanding that the model is a generation layer, not the quality layer.

Responsibilities

• Own and evolve the multi-layered eval pipeline and verification portfolio, including deterministic quality checks and LLM-graded transcript evaluation.
• Advance end-to-end system evaluation across orchestrated agents, RAG-supported responses, and multi-party voice conversations.
• Improve observability to surface eval metrics, detect regressions, and enable data-driven quality decisions.

Skills

QA for AI/ML systems

Python programming

Observability

Communication

Healthcare focus

Tools

pytest

FastAPI TestClient

Pydantic

CircleCI

Job description

Third Way Health is seeking a Senior QA / Eval Engineer to own and evolve the quality and evaluation infrastructure behind our AI-powered patient engagement platform. You will shape how we define quality across automated and manual workflows and build tooling to make that definition measurable and actionable.

You will collaborate with ML engineers, product managers, and operations leads to translate real-world failure modes into automated checks and contribute to the QA pipeline orchestration