Hybrid AI Quality Engineer: Ensure Robust, Trustworthy AI

Aviva; Warszawa; On-site; PLN 70,000 - 110,000; Full time

Benefits offered by this job

Performance Bonus

Private medical care

Hybrid working (Poland)

Job summary

Aviva in Warsaw is seeking an AI Quality Engineer to help ensure the quality, safety and reliability of AI-powered products. The role focuses on evaluating conversational AI, autonomous agents and other AI systems, collaborating with engineers, product teams and business stakeholders to ensure robustness and production readiness.

You will translate business outcomes into evaluation criteria, design datasets and evaluation methods, and communicate findings to both technical and non-technical

Qualifications

• Professional experience in quality engineering, software testing, test automation, data science, software engineering, or related field.
• Interest or experience with AI-powered solutions and processes.

Responsibilities

• Translate business expectations and risks into measurable evaluation criteria.
• Define success and failure criteria for AI systems and evaluate them.
• Design evaluation approaches for conversational and multi-turn AI systems.
• Build and maintain evaluation datasets covering common and high-risk scenarios.
• Assess dataset quality and biases, data leakage and freshness.
• Select appropriate evaluation methods (manual, deterministic, model-based, statistical).
• Analyze errors by category and root cause; differentiate model vs system failures.
• Validate automated evaluators and AI judges against human judgment.
• Identify blind spots and emerging failure modes; create baselines and regression tests.
• Communicate results and residual risks effectively for release decisions.

Skills

Python proficiency

AI quality engineering

Analytical thinking

Problem solving

Communication

Stakeholder management

Tools

Git

Jira

Job description

Aviva in Warsaw is seeking an AI Quality Engineer to help ensure the quality, safety and reliability of AI-powered products. The role focuses on evaluating conversational AI, autonomous agents and other AI systems, collaborating with engineers, product teams and business stakeholders to ensure robustness and production readiness.

You will translate business outcomes into evaluation criteria, design datasets and evaluation methods, and communicate findings to both technical and non-technical