Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank; Dubai; Hybrid; AED 230,000 - 320,000; Full time

Benefits offered by this job

Best in class leave policy

Gender neutral parental leaves

Sponsorship for Industry relevantcerts

Employee Assistance Program

Hospitalization Insurance

Accident and Term life Insurance

Complementary Health screening

Job summary

Deutsche Bank AG in Dubai is seeking a Quality Assurance Engineer to validate AI agentic applications across development and production. You will design, build, and execute test strategies for LLM accuracy, RAG ingestion and attribution, and agent planning with tool use.

The role emphasizes automated testing, AI safety, and continuous quality measurement across releases, with a focus on grounding, prompt engineering, and governance in high-stakes environments.

Qualifications

• 5+ years in software quality assurance and testing.
• Hands-on experience testing GenAI, LLM, or agentic AI.
• Strong automation framework background and utilities.
• Proficient in at least one language: Java or Python.
• Familiar with LLM evaluation tools and evaluation metrics.

Responsibilities

• Design, implement and maintain AI testing strategies and automation frameworks.
• Validate LLM-powered applications for accuracy, grounding, and hallucinations.
• Evaluate autonomous/semi-autonomous AI agents and recovery behaviors.
• Assess RAG performance: ingestion, embedding accuracy, and citations.
• Perform AI safety testing for prompts, data leakage, tool use, and privileges.
• Establish metrics and QA gates across releases and deployments.
• Develop and maintain automated test suites (APIs, UI, integration, RAG).
• Analyze production AI interactions to identify failures and create regression tests.

Skills

QA testing

GenAI testing

Automation frameworks

Java/Python

LLM evaluation

Prompt engineering

Education

Bachelor's degree in CS/Engineering

Tools

Langsmith

Langfuse

RAG tools

Vector DBs

CI/CD

Job description

Role Description

Quality Assurance Engineer to validate AI agentic applications across development and production. The position combines specialized AI testing, automation, safety evaluation, and continuous quality measurement. The engineer is responsible to design, build and execute the test strategy for LLM accuracy, RAG ingestion and attribution, agent planning and tool use, recovery behavior, and production failure patterns

What we’ll offer you

As part of our flexible scheme, here are just some of the benefits that you’ll enjoy

• Best in class leave policyGender neutral parental leaves100% reimbursement under childcare assistance benefit (gender neutral)Sponsorship for Industry relevant certifications and educationEmployee Assistance Program for you and your family membersComprehensive Hospitalization Insurance for you and your dependentsAccident and Term life InsuranceComplementary Health screening for 35 yrs. and above

Your key responsibilities

• Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows
• Develop and implement specialized testing approaches for LLM-powered applications, validating response accuracy, relevance, and grounding, while identifying hallucinations and edge-case behaviours
• Test and evaluate the behaviour of autonomous and semi-autonomous AI agents, ensuring proper planning, tool selection/sequencing, agent-to-agent interactions, and graceful error/timeout recovery
• Validate Retrieval-Augmented Generation (RAG) performance, including document ingestion, chunking, metadata quality, embedding accuracy, and citation/source attribution
• Perform AI safety and security testing to identify vulnerabilities such as prompt injection (direct/indirect), data leakage, unauthorized tool execution, and privilege escalation
• Establish quantitative AI evaluation metrics and quality gates (e.g., classification accuracy, groundedness, hallucination rate, tool-selection accuracy, and cost/latency) to measure performance across releases
• Build and maintain automated test suites (covering APIs, UI, integration, RAG, and agent trajectories) to progressively transition the testing suite toward automated continuous evaluation
• Analyze production AI interactions and failed agent journeys to identify recurring failure patterns, create regression tests from production incidents, and ensure decision auditability

Your skills and experience

Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline5+ years of experience in software quality assurance and testing, with strong knowledge of test planning, functional, integration, regression, and end-to-end testing1–2 years of hands‑on experience specifically testing GenAI, LLM, or Agentic AI applicationsStrong experience with test automation and building automation frameworks and test utilities (rather than relying solely on manual tools)Strong programming ability in at least one language: Java / Kotlin or PythonStrong understanding of generative AI technologies, including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents/agentic workflows, and tool/function callingHands‑on experience with LLM and agent evaluation tools such as Langsmith/Langfuse, Ragas, or equivalent frameworks for dataset-based testing, custom evaluation metrics, experiment tracking, tracing, and continuous regression evaluation

Add On Experience (Nice to Have):

Experience testing API and data integrations, including REST APIs, JSON, SQL, databases, and messaging systems like KafkaGood understanding of CI/CD integration and establishing automated quality gatesExperience working in the financial services industry, with knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controlsExperience with AI/LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologiesHands‑on experience with advanced AI testing capabilities such as prompt injection testing, AI evaluation/evals, and AI observabilityExperience testing distributed systems Proven ability to leverage AI tools to enhance productivity, optimise workflows to solve business problems, while applying critical judgment to ensure responsible and ethical use of data and AI outputs.

How we’ll support you

Training and development to help you excel in your careerCoaching and support from experts in your teamA culture of continuous learning to aid progressionA range of flexible benefits that you can tailor to suit your needs

About us and our teams

Please visit our company website for further information:https://www.db.com/company/company.html We strive for a culture in which we are empowered to excel together every day. This includes acting responsibly, thinking commercially, taking initiative and working collaboratively.Together we share and celebrate the successes of our people. Together we are Deutsche Bank Group.We welcome applications from all people and promote a positive, fair and inclusive work environment.