AI Quality Engineering Lead — Risk & Evaluation

Aviva; Polska; Hybrid; PLN 120,000 - 180,000; Full time

Benefits offered by this job

Performance Bonus

Private medical care

Cafeteria benefits

Employee Shares Plan

Wellbeing Leave

Volunteer time off

Internal referral program

Hybrid work from Poland

Job summary

Aviva Services Excellence Centre in Poland is seeking an experienced AI Quality Engineer to lead the quality engineering approach for AI solutions, from risk assessment to release and monitoring. You will assess AI behaviour, build representative datasets, and ensure safety, bias, and privacy standards while collaborating with product, engineering and risk teams.

The role requires hands-on Python, API/log analysis, and automation design, with experience in CI/CD and observability.

Qualifications

• Several years of relevant professional experience in quality engineering, software testing, software engineering, data science or a related technical field.
• Hands-on experience evaluating AI, large language model, RAG or agent-based applications.
• Practical Python skills and experience with APIs, logs, traces, structured datasets and SQL or similar analysis.
• Ability to design repeatable evaluations and automation, not just manual testing or vendor scoring.
• Good understanding of sampling, thresholds, trends and non-deterministic system behaviour.
• Experience with automated testing, version control, CI/CD and maintainable engineering practices.
• Ability to turn technical findings into decision-quality evidence for business, risk and engineering stakeholders.
• Experience coaching or leading engineers in an emerging or ambiguous domain.
• Strong communication and influencing skills, with the confidence to challenge weak evidence.

Responsibilities

• Lead quality engineering approach for AI solutions from risk assessment through release and monitoring.
• Evaluate multi-turn, RAG and agentic AI behaviour, including grounding, guardrails and tool use.
• Build datasets and assess coverage, balance, leakage, label quality and freshness.
• Assess correctness, relevance, hallucination, safety, bias, privacy and behavioural drift.
• Investigate unexpected outcomes across data, prompts, models, retrieval, integration and evaluators.
• Develop reusable evaluation components, automated test pipelines and evidence-generation patterns.
• Integrate evaluation into CI/CD, observability and engineering workflows with traceability.
• Produce concise, auditable findings that explain confidence, uncertainty and residual risk.
• Partner with product, engineering, data, risk and compliance to influence delivery decisions.

Skills

Quality engineering

AI evaluation

Python

APIs & logs

SQL

Automation design

Testing & CI/CD

Communication

Coaching/leadership

Tools

DeepEval

Arize

Job description

Aviva Services Excellence Centre in Poland is seeking an experienced AI Quality Engineer to lead the quality engineering approach for AI solutions, from risk assessment to release and monitoring. You will assess AI behaviour, build representative datasets, and ensure safety, bias, and privacy standards while collaborating with product, engineering and risk teams.

The role requires hands-on Python, API/log analysis, and automation design, with experience in CI/CD and observability.