Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank AG; Maharashtra; On-site; INR 1,400,000 - 1,900,000; Full time

Benefits offered by this job

Leave policy

Childcare assistance

Certifications sponsorship

Hospitalization Insurance

EAP for family

Health screening

Job summary

Deutsche Bank AG in Pune is seeking a Quality Assurance Engineer to validate AI agentic applications across development and production. The role requires designing, building, and executing test strategies for LLM accuracy, RAG ingestion, and agent tool usage, with focus on safety and recovery behaviors.

You will implement automated test suites for APIs, UI, integration, RAG, and agent trajectories, and analyze production AI interactions to identify recurring failure patterns for regression

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or related discipline.
• 5+ years of software quality assurance and testing experience.
• 1–2 years testing GenAI/LLM/Agentic AI.
• Strong automation framework experience and programming ability in Python/Java.

Responsibilities

• Design, implement, and maintain test strategies and automation for AI agentic workflows.
• Evaluate LLM response accuracy, grounding, and hallucinations; validate tool use sequences.
• Test autonomous/semi-autonomous AI agents for planning, execution, and recovery.
• Assess RAG performance: ingestion, chunking, embeddings, citations, and attribution.
• Establish AI evaluation metrics and automated quality gates; build regression suites.

Skills

QA Testing

Automation Frameworks

GenAI/LLM Testing

Programming: Python/Java

Education

Bachelor's in CS/Engineering

Tools

Langsmith

Ragas

LangChain

Job description

Job Description:Job Title: Quality Assurance Engineer – AI Agentic ApplicationsLocation: Pune, IndiaCorporate Title: AVPRole DescriptionQuality Assurance Engineer to validate AI agentic applications across development and production. The position combines specialized AI testing, automation, safety evaluation, and continuous quality measurement. The engineer is responsible to design, build and execute the test strategy for LLM accuracy, RAG ingestion and attribution, agent planning and tool use, recovery behavior, and production failure patternsWhat we’ll offer youAs part of our flexible scheme, here are just some of the benefits that you’ll enjoyBest in class leave policyGender neutral parental leaves100% reimbursement under childcare assistance benefit (gender neutral)Sponsorship for Industry relevant certifications and educationEmployee Assistance Program for you and your family membersComprehensive Hospitalization Insurance for you and your dependentsAccident and Term life InsuranceComplementary Health screening for 35 yrs. and aboveYour key responsibilitiesDesign, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflowsDevelop and implement specialized testing approaches for LLM-powered applications, validating response accuracy, relevance, and grounding, while identifying hallucinations and edge-case behavioursTest and evaluate the behaviour of autonomous and semi-autonomous AI agents, ensuring proper planning, tool selection/sequencing, agent-to-agent interactions, and graceful error/timeout recoveryValidate Retrieval-Augmented Generation (RAG) performance, including document ingestion, chunking, metadata quality, embedding accuracy, and citation/source attributionPerform AI safety and security testing to identify vulnerabilities such as prompt injection (direct/indirect), data leakage, unauthorized tool execution, and privilege escalationEstablish quantitative AI evaluation metrics and quality gates (e.g., classification accuracy, groundedness, hallucination rate, tool-selection accuracy, and cost/latency) to measure performance across releasesBuild and maintain automated test suites (covering APIs, UI, integration, RAG, and agent trajectories) to progressively transition the testing suite toward automated continuous evaluationAnalyze production AI interactions and failed agent journeys to identify recurring failure patterns, create regression tests from production incidents, and ensure decision auditabilityYour skills and experienceMinimum Required Skills & Experience (Must Have):Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline5+ years of experience in software quality assurance and testing, with strong knowledge of test planning, functional, integration, regression, and end-to-end testing1–2 years of hands-on experience specifically testing GenAI, LLM, or Agentic AI applicationsStrong experience with test automation and building automation frameworks and test utilities (rather than relying solely on manual tools)Strong programming ability in at least one language: Java / Kotlin or PythonStrong understanding of generative AI technologies, including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents/agentic workflows, and tool/function callingHands-on experience with LLM and agent evaluation tools such as Langsmith/Langfuse, Ragas, or equivalent frameworks for dataset-based testing, custom evaluation metrics, experiment tracking, tracing, and continuous regression evaluationAdd On Experience (Nice to Have):Experience testing API and data integrations, including REST APIs, JSON, SQL, databases, and messaging systems like KafkaGood understanding of CI/CD integration and establishing automated quality gatesExperience working in the financial services industry, with knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controlsExperience with AI/LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologiesHands-on experience with advanced AI testing capabilities such as prompt injection testing, AI evaluation/evals, and AI observabilityExperience testing distributed systemsProven ability to leverage AI tools to enhance productivity, optimise workflows to solve business problems, while applying critical judgment to ensure responsible and ethical use of data and AI outputs.How we’ll support youTraining and development to help you excel in your careerCoaching and support from experts in your teamA culture of continuous learning to aid progressionA range of flexible benefits that you can tailor to suit your needsAbout us and our teamsPlease visit our company website for further information:https://www.db.com/company/company.htmlWe strive for a culture in which we are empowered to excel together every day. This includes acting responsibly, thinking commercially, taking initiative and working collaboratively.Together we share and celebrate the successes of our people. Together we are Deutsche Bank Group.We welcome applications from all people and promote a positive, fair and inclusive work environment.