AI Testing Engineer — Python, LLM Evaluation Pipelines

Reply, Inc.; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Reply, Inc. is seeking a mid/senior AI Testing & Engineering Specialist in Greater London to design and evolve a next‑generation testing framework for generative AI agents. You will build evaluation pipelines, synthetic data generation, observability layers, and quality metrics for LLM-based systems.

You will work across AI engineering, data pipelines, and QA to shape standards for AI system validation in production, contributing to architecture and hands‑on implementation with autonomy.

Qualifications

• 2–5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering.
• Strong proficiency in Python and data/AI libraries (pandas, numpy, PyTorch or similar).
• Solid understanding of LLMs, NLP concepts, or generative AI systems.
• Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines.
• Familiarity with data pipelines, dataset design, or feature engineering is a strong plus.

Responsibilities

• Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems.
• Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems.
• Build and maintain tools for test data generation (including synthetic and adversarial datasets).
• Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.).
• Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms.
• Support continuous testing, benchmarking, and production monitoring of AI systems.
• Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability.

Skills

Python

Data libraries

LLM concepts

Test automation

CI/CD

Tools

PyTest

Playwright

CI/CD tooling

Job description

Reply, Inc. is seeking a mid/senior AI Testing & Engineering Specialist in Greater London to design and evolve a next‑generation testing framework for generative AI agents. You will build evaluation pipelines, synthetic data generation, observability layers, and quality metrics for LLM-based systems.

You will work across AI engineering, data pipelines, and QA to shape standards for AI system validation in production, contributing to architecture and hands‑on implementation with autonomy.