Ai Test Engineer

Reply

Piemonte

On-site

GBP 60,000 - 95,000

Full time

11 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Reply is seeking an AI Test Engineer to design and develop Python-based testing frameworks for evaluating AI agents and LLM-based systems. You will help shape evaluation pipelines, observability, and data-driven testing, collaborating closely with AI and data teams.

The role requires 2-5+ years in software/AI engineering, strong Python skills, and experience with PyTest/Playwright, CI/CD, and NLP metrics. This position is based in the United Kingdom with potential for growth and impact.

Qualifications

  • 2-5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering
  • Strong proficiency in Python and data/AI libraries (e.g., pandas, numpy, PyTorch or similar)
  • Solid understanding of LLMs, NLP concepts, or generative AI systems
  • Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines
  • Familiarity with data pipelines, dataset design, or feature engineering is a strong plus
  • Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics, or custom scoring approaches)
  • Ability to design scalable systems and work autonomously on complex technical problems
  • Strong analytical mindset and interest in AI system quality, reliability, and governance

Responsibilities

  • Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems
  • Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems
  • Build and maintain tools for test data generation (including synthetic and adversarial datasets)
  • Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.)
  • Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms
  • Support continuous testing, benchmarking, and production monitoring of AI systems
  • Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability

Skills

Python
Pandas
NumPy
PyTorch
PyTest
Playwright
CI/CD
Test Automation
Architecture design

Tools

PyTest
Playwright

Job description

AI Test Engineer

About Reply: Reply specialises in designing and implementing solutions based on new communication channels and digital media. Structured as a network of highly specialised companies, Reply supports leading European industrial groups in sectors such as Telco & Media, Industry and Services, Banking and Insurance, and Public Administration, helping define and develop business models enabled by new paradigms such as AI, Cloud Computing, Digital Media, and the Internet of Things. Reply’s services include Consulting, System Integration, and Digital Services.


Role Overview

We are looking for a mid/senior-level AI Testing & Engineering Specialist with strong Python expertise and solid experience in AI, Data Engineering, or Test Automation. The role is focused on building and evolving a next-generation testing framework for generative AI agents, including evaluation pipelines, synthetic data generation, observability layers, and quality metrics for LLM-based systems. You will work across AI engineering, data pipelines, and quality automation, contributing both to architecture design and hands-on implementation of scalable testing and evaluation systems. This role is suited for someone who can operate independently, contribute to technical decisions, and help shape standards for AI system validation in production environments.


Responsibilities


  • Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems

  • Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems

  • Build and maintain tools for test data generation (including synthetic and adversarial datasets)

  • Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.)

  • Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms

  • Support continuous testing, benchmarking, and production monitoring of AI systems

  • Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability


About the candidate


  • 2-5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering

  • Strong proficiency in Python and data/AI libraries (e.g., pandas, numpy, PyTorch or similar)

  • Solid understanding of LLMs, NLP concepts, or generative AI systems

  • Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines

  • Familiarity with data pipelines, dataset design, or feature engineering is a strong plus

  • Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics, or custom scoring approaches)

  • Ability to design scalable systems and work autonomously on complex technical problems

  • Strong analytical mindset and interest in AI system quality, reliability, and governance


Reply is an Equal Opportunities Employer and committed to embracing diversity in the workplace. We provide equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type regardless of age, sexual orientation, gender, identity, pregnancy, religion, nationality, ethnic origin, disability, medical history, skin colour, marital status or parental status or any other characteristic protected by the Law. Reply is committed to making sure that our selection methods are fair to everyone. To help you during the recruitment process, please let us know of any Reasonable Adjustments you may need.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Ai Forward Deployed Engineer
Senior Ai Forward Deployed Engineer

Reply • Piemonte

On-site
GBP 90,000 - 130,000
Ai And Machine Learning Engineer
Ai And Machine Learning Engineer

Reply • Piemonte

On-site
GBP 65,000 - 90,000
Graduate Ai Engineer Consultant
Graduate Ai Engineer Consultant

Reply • Piemonte

On-site
GBP 30,000 - 36,000
AI Engineer
AI Engineer

Reply • Turbigo

Hybrid
EUR 40,000 - 60,000
Hands-on experience
Flexible working hours
Free coffee
Genai Agentic System
Genai Agentic System

Reply • Trieste/Trst

On-site
EUR 37,000 - 65,000
Senior Agentic AI Engineer
Senior Agentic AI Engineer

Reply • Piemonte

On-site
EUR 34,000 - 65,000
Structured career path
Continuous learning
Benefits of being a Replyer
AI Testing & Evaluation Engineer (Python)
AI Testing & Evaluation Engineer (Python)

Reply • Piemonte

On-site
GBP 60,000 - 95,000
SENIOR AGENTIC AI ENGINEER
SENIOR AGENTIC AI ENGINEER

Reply, Inc. • Bologna, Padova

Remote
EUR 37,000 - 65,000
Career path development
Competitive compensation
Genai Agentic System
Genai Agentic System

Reply • Puglia

On-site
EUR 37,000 - 65,000
Junior Engineer Gen AI
Junior Engineer Gen AI

Reply • Piemonte

On-site
EUR 26,000 - 39,000