AI Testing & Evaluation Engineer (Python)

Reply

Piemonte

In loco

GBP 60.000 - 95.000

Tempo pieno

11 giorni fa

Una candidatura completa in un minuto

Curriculum e lettera di presentazione personalizzati, pronti da inviare.

Supera i filtri ATS

Descrizione del lavoro

Reply is seeking an AI Test Engineer to design and develop Python-based testing frameworks for evaluating AI agents and LLM-based systems. You will help shape evaluation pipelines, observability, and data-driven testing, collaborating closely with AI and data teams.

The role requires 2-5+ years in software/AI engineering, strong Python skills, and experience with PyTest/Playwright, CI/CD, and NLP metrics. This position is based in the United Kingdom with potential for growth and impact.

Competenze

  • 2-5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering
  • Strong proficiency in Python and data/AI libraries (e.g., pandas, numpy, PyTorch or similar)
  • Solid understanding of LLMs, NLP concepts, or generative AI systems
  • Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines
  • Familiarity with data pipelines, dataset design, or feature engineering is a strong plus
  • Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics, or custom scoring approaches)
  • Ability to design scalable systems and work autonomously on complex technical problems
  • Strong analytical mindset and interest in AI system quality, reliability, and governance

Mansioni

  • Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems
  • Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems
  • Build and maintain tools for test data generation (including synthetic and adversarial datasets)
  • Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.)
  • Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms
  • Support continuous testing, benchmarking, and production monitoring of AI systems
  • Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability

Conoscenze

Python
Pandas
NumPy
PyTorch
PyTest
Playwright
CI/CD
Test Automation
Architecture design

Strumenti

PyTest
Playwright

Descrizione del lavoro

Reply is seeking an AI Test Engineer to design and develop Python-based testing frameworks for evaluating AI agents and LLM-based systems. You will help shape evaluation pipelines, observability, and data-driven testing, collaborating closely with AI and data teams.

The role requires 2-5+ years in software/AI engineering, strong Python skills, and experience with PyTest/Playwright, CI/CD, and NLP metrics. This position is based in the United Kingdom with potential for growth and impact.

Ottieni la revisione del curriculum gratis e riservata.

o trascina qui il file.

Similar jobs

Offerte di lavoro simili che vale la pena confrontare

Ai Test Engineer
Ai Test Engineer

Reply • Piemonte

In loco
GBP 60.000 - 95.000
Senior Python Engineer - AI Testing Project (Freelance, Mindrift)
Senior Python Engineer - AI Testing Project (Freelance, Mindrift)

Mindrift • Italia

Remoto
EUR 45.247 - 73.526
Fully remote and flexible participation
Task-based compensation up to $30/hour
Opportunity to work on innovative AI projects
AI Engineer: Build Scalable AI Platforms & Pipelines
AI Engineer: Build Scalable AI Platforms & Pipelines

TeamSystem • Lombardia

Ibrido
EUR 41.000 - 52.000
Total compensation €41k-€52k
Wellbeing Wallet €2,800/year
Hybrid work model
+1
Evaluation Scenario Writer - AI Agent Testing Specialist
Evaluation Scenario Writer - AI Agent Testing Specialist

Mindrift • Italia

Remoto
EUR 23.674 - 35.511
Flexible, remote working hours
Competitive pay up to $30/hour
Gain valuable experience for your portfolio
AI Coding Agent Evaluator - Design & QA Real-World Tasks
AI Coding Agent Evaluator - Design & QA Real-World Tasks

Mindrift • Roma

Remoto
EUR 6915 - 27.660
Python Developer
Python Developer

Set Europa • Lombardia

Remoto
EUR 46.000 - 70.000
Graduate AI Engineer Consultant - Azure & Python
Graduate AI Engineer Consultant - Azure & Python

Reply • Piemonte

In loco
GBP 30.000 - 36.000
AI Engineer
AI Engineer

TMC • Lombardia

In loco
EUR 65.000 - 90.000
Senior Python AI Developer - LLMs, RAG & AI Agents (Remote)
Senior Python AI Developer - LLMs, RAG & AI Agents (Remote)

Set Europa • Lombardia

Remoto
EUR 46.000 - 70.000
Senior AI Engineer – Flexible/Remote + Training
Senior AI Engineer – Flexible/Remote + Training

msg global solutions • Lombardia

Ibrido
EUR 55.000 - 65.000
Health Insurance
Welfare Plan
Job for Friends
+5