Hybrid AI Quality Engineering Lead

Aviva; Warszawa; Hybrid; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Contract of employment

Performance Bonus

Private medical care

Cafeteria system

Baby leave

Shortened working time

Group insurance

Employee Shares Plan

Extra day off for birthday

Volunteer time off

Additional leave accrual

Wellbeing Leave

Psychological support

E-learning platforms

Employee referral program

Hybrid working Poland

Warsaw office location

Work Without Barriers

Diversity Charter

Job summary

Aviva Services Excellence Centre is seeking an AI Quality Engineering Team Lead to establish scalable testing for GenAI and agentic AI across Aviva solutions. You will combine hands-on evaluation with team leadership to ensure accuracy, safety and release readiness of AI systems.

You will connect business risk with data, evaluating prompts, models and tooling while building auditable findings and reusable evaluation components for multiple AI products.

Qualifications

• Proven experience in quality engineering for AI/ML systems.
• Hands-on evaluation of AI/LLM and agent-based apps.
• Strong Python skills and data analysis ability.
• Design repeatable evaluations and automation, not just tests.
• Good sampling, thresholds and non-deterministic system understanding.
• Experience with automated testing, version control and CI/CD.
• Ability to turn findings into business decisions for risk teams.
• Experience coaching engineers in new domains.
• Excellent communication and influencing skills.

Responsibilities

• Lead the quality engineering approach for AI solutions from risk assessment to release and monitoring.
• Evaluate multi-turn, RAG and agentic AI behaviour including grounding and guardrails.
• Build representative datasets and assess coverage, bias and freshness.
• Investigate correctness, relevance, safety, privacy and drift.
• Identify and resolve unexpected outcomes across data, prompts, models and retrieval.
• Develop reusable evaluation components and automated test pipelines.
• Align evaluators with human judgment and baselines.
• Use metrics and regression tests to support release decisions.
• Integrate evaluation into CI/CD and engineering workflows with traceability.
• Produce concise, auditable findings on confidence and residual risk.
• Collaborate with product, engineering, risk, compliance and data teams.
• Coach AI Quality Engineers and foster collaboration and learning.

Skills

AI QA

QA testing

Python

APIs

Logs

SQL

CI/CD

Data analysis

Leadership

Communication

Education

Job description

Aviva Services Excellence Centre is seeking an AI Quality Engineering Team Lead to establish scalable testing for GenAI and agentic AI across Aviva solutions. You will combine hands-on evaluation with team leadership to ensure accuracy, safety and release readiness of AI systems.

You will connect business risk with data, evaluating prompts, models and tooling while building auditable findings and reusable evaluation components for multiple AI products.