AI Test Automation Engineer - Python & LLM Quality

Reply; England; On-site; GBP 32,000 - 52,000; Full time

Job summary

Reply is seeking a Graduate AI Test Automation Engineer in the United Kingdom to design and implement a Python-based testing framework for generative AI agents. You will develop tools for test data generation, automate evaluation processes, and contribute to model quality metrics with a collaborative, high-impact environment.

The role requires a degree in Computer Science/AI, strong Python skills, and familiarity with testing tools like PyTest and Playwright.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, AI, or related field (2.1 min grade).
• Strong interest in AI-driven software testing and development.
• Proficient in Python and data libraries.
• Familiarity with LLMs and testing tools (PyTest, Playwright) is a plus.

Responsibilities

• Develop and maintain Python-based test automation tools.
• Collaborate on framework design and third-party tool integration.
• Define and implement test strategies for LLM behaviour validation.
• Support continuous testing, benchmarking, and KPI tracking.

Skills

Python

Data libraries

AI testing

Analytical mindset

Education

Bachelor’s or Master’s degree in Computer Science/AI

Tools

PyTest

Playwright

Job description

Reply is seeking a Graduate AI Test Automation Engineer in the United Kingdom to design and implement a Python-based testing framework for generative AI agents. You will develop tools for test data generation, automate evaluation processes, and contribute to model quality metrics with a collaborative, high-impact environment.

The role requires a degree in Computer Science/AI, strong Python skills, and familiarity with testing tools like PyTest and Playwright.