Senior AI Quality Engineer - Automated Testing for LLMs

Xapien; London; On-site; NOK 895,000 - 1,279,000; Full time

Benefits offered by this job

Private health insurance

Life Insurance

Unlimited holidays

£1k PD fund per year

Employee equity

Job summary

Xapien is seeking a senior test automation engineer to help shape the QA backbone of our AI-driven due diligence platform. You will automate, evaluate and improve our evaluation harnesses, datasets and signals like groundedness and accuracy.

You’ll work with Python, Playwright and pytest in a CI/CD environment to deliver robust, auditable test outcomes for non‑deterministic AI outputs. Join a fast-growing team with a strong development culture, supportive leadership, and a focus on real‑world

Qualifications

• Strong test‑automation engineering in a modern stack.
• Proficient in Python and testing frameworks (Playwright/pytest).
• Experience with CI/CD pipelines and test infrastructure.

Responsibilities

• Enhance and run layered eval harnesses including automated checks and LLM‑as‑judge scoring.
• Maintain versioned golden datasets for reproducible evaluation.
• Track signals like groundedness and entity resolution for investigative output.
• Run and improve a tiered CI model built on Python, Playwright and pytest.
• Investigate flaky tests and fix root causes.

Skills

Test automation

Python

Playwright

pytest

CI/CD

Tools

Git

Job description

Xapien is seeking a senior test automation engineer to help shape the QA backbone of our AI-driven due diligence platform. You will automate, evaluate and improve our evaluation harnesses, datasets and signals like groundedness and accuracy.

You’ll work with Python, Playwright and pytest in a CI/CD environment to deliver robust, auditable test outcomes for non‑deterministic AI outputs. Join a fast-growing team with a strong development culture, supportive leadership, and a focus on real‑world