AI Quality Engineer — Ensure Trusted AI Systems

Aviva plc; Warszawa; Hybrid; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Performance Bonus

Private medical care

Cafeteria benefits (MultiSport)

Employee Shares Plan

Hybrid working model in Poland

Job summary

Aviva plc is looking for an AI Quality Engineer to join the Aviva Services Excellence Centre in Warsaw. You will ensure the quality, safety and reliability of AI-powered products, building trust through robust evaluation across conversational and autonomous AI systems.

You will collaborate with engineers, product teams and business stakeholders to define measurable criteria and drive production readiness. The role emphasizes translating business objectives into practical evaluation approaches,

Qualifications

• Professional experience in quality engineering, software testing, test automation, data science or a related field (we can consider different seniority levels).
• Experience or strong interest in working with AI-powered solutions, either through developing, testing, evaluating or supporting them.
• Good Python proficiency and experience writing clean, maintainable code for analysis, automation or testing purposes.
• Interest in understanding how AI systems behave, how reliable they are, and how their quality can be measured.
• Strong analytical and problem-solving skills, with the ability to investigate issues and identify root causes.
• Ability to translate business objectives, customer outcomes and risks into practical evaluation approaches.
• Strong communication skills and the ability to explain findings to both technical and non-technical stakeholders.
• A curious, proactive mindset and a genuine interest in AI, emerging technologies and system robustness.

Responsibilities

• Translate business expectations and risks into measurable evaluation criteria.
• Define what successful and failed system behaviour looks like across AI systems.
• Design evaluation approaches for conversational, multi-turn and non-deterministic AI.
• Build and maintain representative evaluation datasets including high-risk scenarios.
• Assess dataset quality: coverage, balance, leakage and freshness.
• Select manual, deterministic, model-based and statistical evaluation methods.
• Analyze errors by category, journey, risk level and root cause.
• Distinguish model failures from retrieval, ranking, prompts, tools and data issues.
• Validate automated evaluators against human judgement.
• Detect bias and blind spots, and ensure robust evaluation results.
• Create baselines, comparisons and regression tests for version control.
• Prioritize tests by failure probability and impact, not coverage alone.
• Communicate results and residual risks to support release decisions.
• Identify emerging failure modes from changes and production observations.

Skills

Python

AI quality engineering

Software testing

Data science

Problem solving

Communication

Job description

Aviva plc is looking for an AI Quality Engineer to join the Aviva Services Excellence Centre in Warsaw. You will ensure the quality, safety and reliability of AI-powered products, building trust through robust evaluation across conversational and autonomous AI systems.

You will collaborate with engineers, product teams and business stakeholders to define measurable criteria and drive production readiness. The role emphasizes translating business objectives into practical evaluation approaches,