AI Quality Engineer: Build Trustworthy, Production-Ready AI

Aviva; Województwo mazowieckie; Hybrid; PLN 140,000 - 210,000; Full time

Benefits offered by this job

Performance Bonus

Private medical care

Hybrid working model

Employee Shares Plan

Extra time off for special occasions

Job summary

Aviva Services Excellence Centre in Warsaw is seeking an AI Quality Engineer to ensure the quality, safety and reliability of AI-powered products. You will translate business expectations into measurable evaluation criteria, design evaluation methods for conversational AI, build datasets, and analyse errors across systems.

You will collaborate with engineers, product teams and stakeholders, validate automated evaluators, and help drive production readiness for AI solutions while maintaining

Qualifications

• Professional experience in quality engineering, software testing, data science or related field.
• Experience or strong interest in AI-powered solutions.
• Good Python proficiency and clean code for analysis/testing.
• Ability to translate business objectives into evaluation approaches.
• Strong analytical and problem-solving skills.
• Strong communication and stakeholder management skills.
• Curious, proactive mindset with interest in AI and system robustness.

Responsibilities

• Translate business expectations into measurable evaluation criteria.
• Define successful vs. partial/unsuccessful system behavior.
• Design evaluation approaches for conversational, multi-turn AI systems.
• Build and maintain representative evaluation datasets.
• Assess dataset quality and potential data issues.
• Select manual, deterministic, model-based and statistical evaluation methods.
• Analyse errors by category, journey, risk and root cause.
• Distinguish model failures from retrieval, prompts, tools or data issues.
• Validate automated evaluators against human judgment.
• Detect biases, noise and misleading aggregation in results.
• Create baselines, comparisons and regression tests.
• Prioritise tests by risk and impact.
• Communicate results and residual risks for release decisions.

Skills

Python

Quality engineering

Data analysis

AI/ML testing

Test automation

Communication

Git

Jenkins

JSON/log analytics

Tools

Git

Jenkins

JSON/Logs

Notebook tools

Job description

Aviva Services Excellence Centre in Warsaw is seeking an AI Quality Engineer to ensure the quality, safety and reliability of AI-powered products. You will translate business expectations into measurable evaluation criteria, design evaluation methods for conversational AI, build datasets, and analyse errors across systems.

You will collaborate with engineers, product teams and stakeholders, validate automated evaluators, and help drive production readiness for AI solutions while maintaining