AI Test Engineer - Healthcare Platform Automation

ITAC Solutions, LLC; Birmingham; On-site; GBP 105,000 - 120,000; Full time

Benefits offered by this job

Direct impact on patient safety

Test ownership of AI platform

Top-tier health coverage

401(k)

Paid time off and parental leave

Job summary

ITAC Solutions, LLC is seeking a testing lead for an AI-driven healthcare technology platform. You will own the testing strategy and automation, ensuring reliability before any live patient interaction, including non-deterministic conversations and HIPAA-aware testing.

The role emphasizes building scalable test infrastructure, synthetic data generation, and driving quality improvements across engineering teams, with strong emphasis on CI/CD and automation.

Qualifications

• Experience testing AI-powered applications, incl. LLM prompt/model regression testing.
• Experience testing non-deterministic conversational flows with tolerant assertions.
• Experience generating synthetic test data at scale.
• Built test automation frameworks from the ground up.

Responsibilities

• Design, build, and maintain automated test infrastructure (unit, integration, E2E, load).
• Build evaluation/regression harnesses scoring AI behavior by outcome.
• Generate synthetic data and de-identified scenarios for edge cases.
• Create testing frameworks/tools for reliable tests across teams.
• Define and enforce quality standards across engineering teams.
• Own test environments and CI/CD integration for automated runs.
• Triage production quality issues and drive root-cause analysis with HIPAA-aware testing.

Skills

AI testing

LLM prompt testing

Non-deterministic convo testing

Synthetic data generation

Test automation frameworks

Python

TypeScript

CI/CD (GitHub Actions/Jenkins)

Quality improvement

Tools

GitHub Actions

Jenkins

Job description

ITAC Solutions, LLC is seeking a testing lead for an AI-driven healthcare technology platform. You will own the testing strategy and automation, ensuring reliability before any live patient interaction, including non-deterministic conversations and HIPAA-aware testing.

The role emphasizes building scalable test infrastructure, synthetic data generation, and driving quality improvements across engineering teams, with strong emphasis on CI/CD and automation.