AI Testing & Validation Engineer for LLM Systems

Reply; Greater London; On-site; GBP 65,000 - 90,000; Full time

Job summary

Reply seeks a mid/senior-level AI Testing & Engineering Specialist to design and implement Python-based testing frameworks for AI agents and LLM-based systems. You will shape evaluation pipelines, data-driven testing, and observability layers while collaborating with AI engineers, data teams, and QA to ensure reliable scalable testing in production.

You should have 2–5+ years in software/AI engineering, strong Python skills, and familiarity with NLP/LLM concepts and CI/CD practices.

Qualifications

• 2–5+ years in Software Engineering, Test Automation, AI Engineering, or Data Engineering.
• Strong Python proficiency and experience with data/AI libraries such as pandas, numpy, PyTorch.
• Solid understanding of LLMs, NLP concepts, or generative AI systems.

Responsibilities

• Design and develop Python-based frameworks for testing AI agents and LLM systems.
• Contribute to architecture for evaluation pipelines, observability, and data-driven testing.
• Build tools for test data generation, including synthetic and adversarial datasets.

Skills

Python

Test automation

LLM concepts

CI/CD

Tools

PyTorch

Pandas

NumPy

Playwright

Job description

Reply seeks a mid/senior-level AI Testing & Engineering Specialist to design and implement Python-based testing frameworks for AI agents and LLM-based systems. You will shape evaluation pipelines, data-driven testing, and observability layers while collaborating with AI engineers, data teams, and QA to ensure reliable scalable testing in production.

You should have 2–5+ years in software/AI engineering, strong Python skills, and familiarity with NLP/LLM concepts and CI/CD practices.