AI Quality Engineer

RAKBANK; Dubai; On-site; AED 250,000 - 520,000; Full time

Job summary

RAKBANK is seeking an experienced Assistant Manager – Automation and AI Engineer to lead quality engineering across AI, Generative AI, Conversational AI and Intelligent Automation initiatives. This hands‑on role ensures AI-powered solutions are accurate, secure, and production‑ready while driving automation excellence.

You will define AI testing strategies, evaluate LLMs, design testing frameworks, and mentor teams across Product, Engineering and Data Science.

Qualifications

• Bachelor's degree required with 8–10 years of testing experience.
• Experience leading QA/automation in banking tech projects preferred.
• Strong expertise in Generative AI, Conversational AI, and AI quality assurance methodologies.

Responsibilities

• Lead end-to-end quality engineering for AI and automation solutions across Generative AI, Conversational AI, and intelligent automation.
• Define AI testing strategies, evaluation frameworks, and release-readiness criteria.
• Evaluate LLMs for accuracy, safety, latency, and cost; develop RAG evaluation frameworks.
• Design AI testing frameworks covering prompt validation and model regression.
• Build automated testing solutions using Python, PyTest, Playwright, API automation, and CI/CD gates.
• Mentor QA and automation teams and collaborate with Product, Engineering, and Data Science.

Skills

AI testing

Python

PyTest

Playwright

API automation

CI/CD

Azure DevOps

Jenkins

GitLab

Git

Mentoring

Stakeholder mgmt

Education

Bachelor's degree

Tools

UiPath Test Suite

Azure DevOps

Jenkins

GitLab

Git

Playwright

Python

Job description

Rakbank is seeking an experienced Assistant Manager – Automation and AI Engineer to lead quality engineering and testing across our AI, Generative AI, Conversational AI, Agentic AI, and Intelligent Automation initiatives. This hands‑on role will play a critical part in ensuring AI-powered solutions are accurate, reliable, secure, explainable, and production‑ready while driving automation excellence across the Digital & AI ecosystem.

What You Will Do:

• Lead end-to-end quality engineering for Generative AI, Conversational AI, Agentic AI, and intelligent automation solutions.
• Define AI testing strategies, evaluation frameworks, quality standards, and release‑readiness criteria.
• Evaluate and benchmark Large Language Models (LLMs) for accuracy, safety, relevance, latency, groundedness, and cost.
• Design and implement AI testing frameworks covering RAG evaluation, hallucination detection, AI safety, prompt validation, and model regression.
• Build and integrate automated testing solutions using Python, PyTest, Playwright, API automation, and CI/CD quality gates.
• Drive chatbot, virtual assistant, and copilot testing across multi‑turn conversations, intent coverage, and English/Arabic user journeys.
• Mentor QA and automation teams while partnering with Product, Engineering, and Data Science teams to embed quality throughout the AI development lifecycle.

What You Will Bring:

• Bachelors Degree plus 8-10 years of testing experience, including at least 3 years supporting banking technology projects in a lead or SME capacity.
• Strong expertise in Generative AI, Conversational AI, Agentic AI testing, LLM evaluation, RAG frameworks, and AI quality assurance methodologies.
• Hands‑on experience with Python, PyTest, Playwright, API automation, UiPath Test Suite, and intelligent automation testing.
• Knowledge of AI safety testing, prompt injection, jailbreak testing, hallucination detection, privacy controls, and guardrail validation.
• Experience integrating automated testing and AI evaluation into Azure DevOps, Jenkins, GitLab, and modern CI/CD pipelines.
• Strong analytical, stakeholder management, problem‑solving, and mentoring capabilities, with knowledge of banking regulations, UAE PDPL, and digital banking environments.

Preferred Skills:

• Experience in Generative AI, Conversational AI, Agentic AI, and intelligent automation testing.
• Strong expertise in LLM evaluation, benchmarking, prompt validation, model regression, and RAG evaluation frameworks.
• Hands‑on experience in AI quality assurance, including hallucination detection, LLM-as-a-Judge frameworks, and AI safety testing.
• Proficiency in Python, PyTest, Playwright, API automation, and UiPath Test Suite for automation and RPA testing.
• Experience integrating automated testing into CI/CD pipelines using Azure DevOps, Jenkins, GitLab, and Git.
• Knowledge of test data management, synthetic data generation, performance testing, latency profiling, and AI observability.
• Strong understanding of Azure DevOps Test Plans, Jira, and modern AI concepts such as embeddings, vector databases, context windows, and orchestration frameworks.
• Banking domain knowledge with awareness of UAE PDPL, data privacy, security, and regulatory compliance requirements.