Senior QA Engineer - AI Eval & Quality Infra

Third Way Health, Inc.; Cambridge (MA); On-site; USD 120,000 - 150,000; Full time

Job summary

A healthcare innovative company is seeking a Senior QA / Eval Engineer to enhance evaluation infrastructures for their AI-powered patient engagement platform. The role involves extending the evaluation pipeline, driving improvements to metrics, and ensuring quality in interactions. Ideal candidates will possess 5+ years in software engineering, proficiency in Python, and a strong understanding of quality systems in AI, particularly in a healthcare setting. Communication skills and a passion for healthcare innovation are essential.

Qualifications

• 5+ years of software engineering or test engineering experience.
• Strong proficiency in Python for building test frameworks.
• Solid understanding of voice AI or conversational AI systems.
• Excellent communication skills for collaboration.
• Strong interest in healthcare innovation.

Responsibilities

• Own and extend multi-layered eval pipeline and verification portfolio.
• Drive improvements to observability stack for eval metrics.
• Partner with ML engineers for automated checks based on real-world incidents.
• Build real-time monitoring and verification loops.

Skills

Software engineering or test engineering experience

Python programming

Designing evaluation or verification systems

Understanding of voice AI or conversational AI

Communication skills

Tools

pytest

FastAPI TestClient

Pydantic

Job description

A healthcare innovative company is seeking a Senior QA / Eval Engineer to enhance evaluation infrastructures for their AI-powered patient engagement platform. The role involves extending the evaluation pipeline, driving improvements to metrics, and ensuring quality in interactions. Ideal candidates will possess 5+ years in software engineering, proficiency in Python, and a strong understanding of quality systems in AI, particularly in a healthcare setting. Communication skills and a passion for healthcare innovation are essential.