Quality Assurance Engineer (Automation)

present of AI; United Kingdom; Remote; GBP 60,000 - 90,000; Full time

Job summary

Lyrebird Health seeks a hands-on QA engineer focused on automated testing for an AI clinical software product. You will write test code in TypeScript and Python, ensuring fast and reliable Playwright and API test suites, and you will make evidence-based release decisions.

The role emphasizes evaluating model outputs for clinical trust, using tools like DeepEval and Langfuse traces, and designing robust evaluations for non-deterministic responses. Experience with CI pipelines and Docker is valued.

Qualifications

• Hands-on testing focus for automated AI software.
• Write test code in TypeScript and Python.
• Own the final release decision based on evidence.

Responsibilities

• Develop and maintain automated test suites in TS/Python.
• Keep Playwright and API tests fast and reliable.
• Evaluate AI model outputs for clinical safety and trust.

Skills

TS testing

Python testing

CI pipelines

Docker

OpenAPI

Tools

Playwright

DeepEval

Langfuse traces

Job description

This is a hands on engineering job focused on automated testing, not manual checking. A strong candidate writes test code in TypeScript and Python, keeps Playwright and API suites fast and reliable, and can judge whether AI model output is accurate enough for clinicians. The role also owns the final release decision, so clear evidence based judgement matters as much as coding skill.

How it fits the AI era

The product itself is an AI scribe, so the work sits close to model output that changes from run to run. Much of the testing craft stays human: deciding what counts as correct clinical output, designing evaluations when exact matches fail, and making the call on whether a release is safe. AI tools speed up test writing and monitoring, but the judgement about clinical trust remains with the engineer.

How it builds AI experience

The role offers direct practice in evaluating large language model output using tools like DeepEval, openevals and Langfuse traces, which is a rare and growing skill. Building this means learning to design evaluations for non deterministic responses and reading traces to spot where output drifts. Experience with contract tests and load testing adds a solid base for testing AI features in a clinical product.

Worth showing

• TypeScript and Python testing
• API contract testing OpenAPI
• CI pipelines and Docker

This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.

About Lyrebird Health

An AI scribe built directly into Australia's most-used GP software, auto-filling care plans from the consultation rather than just writing a note.