Generative AI Testing Engineer - Python & Validation

Reply, Inc.; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Reply is seeking a mid/senior AI Testing & Engineering Specialist with strong Python expertise to build and evolve a next-generation testing framework for generative AI agents. You will work across AI engineering, data pipelines, and quality automation, shaping standards for AI system validation in production environments.

The role focuses on architecture design and hands-on development of evaluation pipelines, synthetic data generation, and observability layers to deliver robust quality

Qualifications

• 2–5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering.
• Strong proficiency in Python and data/AI libraries (Pandas, NumPy, PyTorch).
• Experience with test automation frameworks (PyTest, Playwright) and CI/CD pipelines.
• Familiarity with data pipelines, dataset design, or feature engineering is a strong plus.
• Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics) preferred.

Responsibilities

• Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems
• Contribute to architecture design for evaluation pipelines, observability, and data‑driven testing systems
• Build and maintain tools for test data generation (including synthetic and adversarial datasets)
• Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.)
• Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms
• Support continuous testing, benchmarking, and production monitoring of AI systems
• Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability

Skills

Python

AI Testing

Test Automation

Data Engineering

LLM / NLP

Tools

PyTest

Playwright

CI/CD

Pandas

NumPy

PyTorch

Job description

Reply is seeking a mid/senior AI Testing & Engineering Specialist with strong Python expertise to build and evolve a next-generation testing framework for generative AI agents. You will work across AI engineering, data pipelines, and quality automation, shaping standards for AI system validation in production environments.

The role focuses on architecture design and hands-on development of evaluation pipelines, synthetic data generation, and observability layers to deliver robust quality