AI QA Engineer

WeDo Technology Solutions Limited; Greater London; Hybrid; GBP 70,000 - 95,000; Full time

Benefits offered by this job

10% bonus

Employee shares/equity

Private healthcare

Life insurance

Unlimited holiday

£1,000 annual PD fund

Job summary

WeDo Technology Solutions Limited is hiring an AI QA Engineer in London. The role sits in a hybrid setup, with typically 2 in-office days weekly, and focuses on building evaluation systems, automation, and tooling to scale quality across engineering squads.

You will work on non-deterministic AI outputs, with a strong emphasis on automated checks, LLM-evaluation rubrics, and robust datasets. The package includes a £70K-£95K base, 10% bonus and comprehensive benefits.

Qualifications

• Experience in AI QA, testing or LLM evaluation.
• Hands-on experience testing non-deterministic AI/LLM outputs.
• Strong test automation engineering experience.
• Familiarity with LLM evaluation methods (golden datasets, rubrics, LLM-as-judge, regression evaluation).
• Proficiency in Python or another OO language.
• Experience with modern automation frameworks (Playwright, pytest).
• Strong understanding of CI/CD and scalable testing infrastructure.
• Analytical and sceptical mindset to QA outcomes.
• Excellent communication to explain complex quality problems to diverse audiences.
• Ability to work autonomously while collaborating across squads.
• Eligible to work in the UK (sponsorship not available).

Responsibilities

• Enhance and maintain AI evaluation frameworks, including automated checks and scoring.
• Maintain versioned golden datasets for reproducibility and auditability.
• Track and improve AI quality across groundedness, hallucination rates, entity resolution and source quality.
• Own and improve test automation and tooling across the product.
• Work with Python, Playwright and pytest across existing test infra.
• Investigate flaky tests and evolving evaluation metrics to identify root causes.
• Collaborate with engineering and product teams to improve testing practices.
• Build tooling and share knowledge enabling engineers to own quality.

Skills

AI QA

AI testing

LLM evaluation

Test automation engineering

RL/IR evaluation methods

Python

Playwright

pytest

CI/CD

Quality metrics

stakeholder communication

autonomous working

UK work rights

Tools

Python

Playwright

pytest

CI/CD

Job description

Job Title: AI QA Engineer

Salary: £70K-£95K base+ 10% bonus + benefits

Location: Moorgate, London – Hybrid, typically 2 days per week in the office

Work Type: Permanent

Role

We’re working with a fast-growing, AI-first technology business that is building an innovative product using cutting-edge AI.

As the business continues to scale, they’re looking for an AI QA Engineer to help solve one of the biggest challenges when building AI products: how do you confidently test and ship non-deterministic AI output at scale?

You’ll join a small QA function with a big influence across engineering. This isn’t a traditional QA environment where testing sits solely with the QA team. You’ll build the evaluation systems, automation and tooling that allow quality to scale across multiple engineering squads.

You’ll be joining an environment where AI is already central to the product and the way the engineering team works, giving you the opportunity to tackle real AI quality problems and influence how the function develops as the company grows.

Responsibilities

• Enhance and maintain AI evaluation frameworks, including automated checks, LLM-as-judge scoring, rubrics and targeted human review
• Maintain versioned golden datasets to keep AI evaluation reproducible and auditable
• Track and improve AI quality across groundedness, hallucination rates, entity resolution and source quality
• Own and improve test automation and tooling across the product
• Work with Python, Playwright and pytest across the existing test infrastructure
• Investigate flaky tests and changing evaluation metrics, identifying and resolving problems at their root cause
• Work closely with engineering and product teams to improve testing practices across different squads
• Build tooling and share knowledge that enables engineers to take greater ownership of quality

Required Skills

• Strong experience within AI QA, AI testing or LLM evaluation
• Hands-on experience evaluating and testing non-deterministic AI/LLM outputs
• Strong test automation engineering experience
• Experience with LLM evaluation techniques such as golden datasets, rubric design, LLM-as-judge or regression evaluation
• Experience with Python or another object-oriented programming language
• Experience with modern automation frameworks such as Playwright, pytest or similar
• Strong understanding of CI/CD and building automated testing as scalable infrastructure
• An analytical and sceptical approach to quality, with the ability to recognise when a successful test result doesn’t necessarily mean the output is correct
• Strong communication skills with the ability to explain complex quality problems to both technical and non-technical stakeholders
• Comfortable working autonomously while influencing and collaborating with engineers across multiple squads
• Must have the right to work in the UK – sponsorship is not available

Why should I apply?

This is an opportunity to work on AI quality problems that go far beyond traditional functional testing.

You’ll be testing non-deterministic AI systems, improving evaluation approaches and helping determine whether AI-generated outputs are genuinely reliable rather than simply relying on a passing test.

You’ll also have genuine influence. The QA function works across engineering, so the tooling, frameworks and approaches you build will help shape how multiple squads think about and deliver quality.

You’ll be joining a genuinely AI-first environment, working alongside people who are continually experimenting with how AI can improve both the product and the way they work.

Alongside a 10% bonus, you’ll receive employee shares/equity, private healthcare, life insurance, unlimited holiday and a £1,000 annual professional development fund.