AI QA Engineer

WeDoTech; Greater London; Hybrid; GBP 70,000 - 95,000; Full time

Benefits offered by this job

Employee shares/equity

Private healthcare

Life insurance

Unlimited holiday

£1,000 annual development fund

Job summary

WeDoTech in London is seeking an AI QA Engineer to help test non-deterministic AI outputs at scale. You will join a small QA function with influence across engineering, building evaluation systems, automation and tooling.

You will work with Python, Playwright and pytest, improving AI quality across groundedness, hallucinations and source quality, while collaborating with multiple squads and owning testing practices.

Qualifications

• Experience in AI QA, AI testing, or LLM evaluation.
• Hands-on testing of non-deterministic AI/LLM outputs.
• Strong test automation engineering experience.

Responsibilities

• Enhance AI evaluation frameworks with automated checks.
• Maintain versioned golden data sets for reproducibility.
• Track AI quality across groundedness, hallucinations, entity resolution.
• Own and improve test automation and tooling.
• Work with Python, Playwright and pytest across tests.
• Investigate flaky tests and root causes.
• Collaborate with squads to improve testing practices.
• Build tooling to empower engineers on quality ownership.

Skills

AI QA

LLM evaluation

Test automation

Python

CI/CD

Analytical thinking

Communication

Autonomous working

Tools

Playwright

pytest

CI/CD pipelines

Job description

Job Title: AI QA Engineer

Salary: £70K-£95K base+ 10% bonus + benefits

Location: Moorgate, London – Hybrid, typically 2 days per week in the office

Work Type: Permanent

Role

We’re working with a fast-growing, AI-first technology business that is building an innovative product using cutting-edge AI.

As the business continues to scale, they’re looking for an AI QA Engineer to help solve one of the biggest challenges when building AI products: how do you confidently test and ship non-deterministic AI output at scale?

You’ll join a small QA function with a big influence across engineering. This isn’t a traditional QA environment where testing sits solely with the QA team. You’ll build the evaluation systems, automation and tooling that allow quality to scale across multiple engineering squads.

You’ll be joining an environment where AI is already central to the product and the way the engineering team works, giving you the opportunity to tackle real AI quality problems and influence how the function develops as the company grows.

Responsibilities

• Enhance and maintain AI evaluation frameworks, including automated checks, LLM-as-judge scoring, rubrics and targeted human review
• Maintain versioned golden datasets to keep AI evaluation reproducible and auditable
• Track and improve AI quality across groundedness, hallucination rates, entity resolution and source quality
• Own and improve test automation and tooling across the product
• Work with Python, Playwright and pytest across the existing test infrastructure
• Investigate flaky tests and changing evaluation metrics, identifying and resolving problems at their root cause
• Work closely with engineering and product teams to improve testing practices across different squads
• Build tooling and share knowledge that enables engineers to take greater ownership of quality

Required Skills

• Strong experience within AI QA, AI testing or LLM evaluation
• Hands‑on experience evaluating and testing non-deterministic AI/LLM outputs
• Strong test automation engineering experience
• Experience with LLM evaluation techniques such as golden datasets, rubric design, LLM-as-judge or regression evaluation
• Experience with Python or another object-oriented programming language
• Experience with modern automation frameworks such as Playwright, pytest or similar
• Strong understanding of CI/CD and building automated testing as scalable infrastructure
• An analytical and sceptical approach to quality, with the ability to recognise when a successful test result doesn’t necessarily mean the output is correct
• Strong communication skills with the ability to explain complex quality problems to both technical and non-technical stakeholders
• Comfortable working autonomously while influencing and collaborating with engineers across multiple squads
• Must have the right to work in the UK – sponsorship is not available

Why should I apply?

This is an opportunity to work on AI quality problems that go far beyond traditional functional testing.

You’ll be testing non-deterministic AI systems, improving evaluation approaches and helping determine whether AI-generated outputs are genuinely reliable rather than simply relying on a passing test.

You’ll also have genuine influence. The QA function works across engineering, so the tooling, frameworks and approaches you build will help shape how multiple squads think about and deliver quality.

You’ll be joining a genuinely AI-first environment, working alongside people who are continually experimenting with how AI can improve both the product and the way they work.

Alongside a 10% bonus, you’ll receive employee shares/equity, private healthcare, life insurance, unlimited holiday and a £1,000 annual professional development fund.