Automation Test Engineer

ITAC Solutions, LLC; Birmingham; On-site; GBP 105,000 - 120,000; Full time

Benefits offered by this job

Direct impact on patient safety

Test ownership of AI platform

Top-tier health coverage

401(k)

Paid time off and parental leave

Job summary

ITAC Solutions, LLC is seeking a testing lead for an AI-driven healthcare technology platform. You will own the testing strategy and automation, ensuring reliability before any live patient interaction, including non-deterministic conversations and HIPAA-aware testing.

The role emphasizes building scalable test infrastructure, synthetic data generation, and driving quality improvements across engineering teams, with strong emphasis on CI/CD and automation.

Qualifications

• Experience testing AI-powered applications, incl. LLM prompt/model regression testing.
• Experience testing non-deterministic conversational flows with tolerant assertions.
• Experience generating synthetic test data at scale.
• Built test automation frameworks from the ground up.

Responsibilities

• Design, build, and maintain automated test infrastructure (unit, integration, E2E, load).
• Build evaluation/regression harnesses scoring AI behavior by outcome.
• Generate synthetic data and de-identified scenarios for edge cases.
• Create testing frameworks/tools for reliable tests across teams.
• Define and enforce quality standards across engineering teams.
• Own test environments and CI/CD integration for automated runs.
• Triage production quality issues and drive root-cause analysis with HIPAA-aware testing.

Skills

AI testing

LLM prompt testing

Non-deterministic convo testing

Synthetic data generation

Test automation frameworks

Python

TypeScript

CI/CD (GitHub Actions/Jenkins)

Quality improvement

Tools

GitHub Actions

Jenkins

Job description

Our client is a healthcare technology company where software quality directly affects whether patients get the care and access they need. You’ll own the testing strategy and automation for an AI-driven platform, making sure every behavior and integration works as expected before it ever reaches a live patient. It’s a rare chance to solve testing problems where the system never gives the same answer twice.

C2C is not an option with this job opening, and all applicants must be able to work for any U.S. employer without sponsorship. Sponsorship is not provided and will not be required in the future.

Benefits & Extras

• Direct impact on patient safety and access to care.
• Ownership of test strategy and infrastructure for a cutting-edge AI platform.
• Top-tier health coverage: medical, dental, and vision.
• 401(k).
• Paid time off, holidays, and paid parental leave.

Compensation

• $150,000

What You’ll Be Doing

• Design, build, and maintain automated test infrastructure covering unit, integration, end-to-end, and load testing.
• Build evaluation and regression harnesses that score AI behavior by outcome, not exact output.
• Generate synthetic conversation data and de-identified patient scenarios to cover rare edge cases.
• Build testing frameworks and tooling that help the wider engineering team write reliable tests.
• Define and enforce quality standards and best practices across engineering teams.
• Own test environments and CI/CD integration for automated test runs.
• Triage production quality issues and drive root cause analysis, including HIPAA and compliance-aware testing.

What You’ll Need to be Considered

• Experience testing AI-powered applications, including LLM prompt and model regression testing.
• Experience testing non-deterministic conversational flows, with assertions that tolerate variation without hiding real regressions.
• Experience generating synthetic test data at scale.
• Built test automation frameworks from the ground up.
• Python or TypeScript skills, with CI/CD experience such as GitHub Actions or Jenkins.
• Track record of raising quality standards across a team.