AI Testing & Automation Engineer (Python/LLM Evaluation)

Reply; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

Reply is seeking a mid/senior-level AI Testing & Engineering Specialist with strong Python expertise to build a next-generation testing framework for generative AI agents, including evaluation pipelines, synthetic data, observability, and quality metrics for LLM-based systems.

You will collaborate across AI engineering, data pipelines, and QA to design scalable testing infrastructure and contribute to standards for production AI validation.

Qualifications

• 2–5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering.
• Strong Python and data/AI libraries experience (e.g., pandas, numpy, PyTorch or similar).
• Familiarity with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics).

Responsibilities

• Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems.
• Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems.
• Build and maintain tools for test data generation (synthetic and adversarial datasets).
• Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate).
• Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms.
• Support continuous testing, benchmarking, and production monitoring of AI systems.
• Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability.

Skills

Python

AI Engineering

Test Automation

Data Engineering

CI/CD pipelines

PyTest

Playwright

LLM evaluation

Tools

PyTest

Playwright

Job description

Reply is seeking a mid/senior-level AI Testing & Engineering Specialist with strong Python expertise to build a next-generation testing framework for generative AI agents, including evaluation pipelines, synthetic data, observability, and quality metrics for LLM-based systems.

You will collaborate across AI engineering, data pipelines, and QA to design scalable testing infrastructure and contribute to standards for production AI validation.