Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank AG; Pune District; Presencial; INR 1 200 000 - 1 800 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Leave policy

Parental leaves

Certification support

Employee Assistance Program

Hospitalization Insurance

Life Insurance

Health screening

Resumo da oferta

Deutsche Bank AG in Pune is seeking a Quality Assurance Engineer to validate AI agentic applications across development and production. You will design, build, and execute test strategies for LLM accuracy, RAG ingestion, and agent planning, with focus on safety, grounding, and error handling.

The role requires hands-on experience testing GenAI/LLMs, building automation frameworks, and strong programming skills in Java or Python.

Qualificações

• Bachelor's degree or above in Computer Science, Engineering, IT, or related.
• 5+ years in software quality assurance with test planning and end-to-end testing.
• 1–2 years specifically testing GenAI/LLM/Agentic AI applications.
• Strong automation experience and building test frameworks.

Responsabilidades

• Design, execute, and maintain test strategies and automation for AI-powered workflows.
• Develop specialized testing approaches for LLM-powered apps validating accuracy and grounding.
• Test autonomous agent behavior including planning, tool use, and recovery.
• Validate RAG performance: ingestion, chunking, metadata, embeddings, citations.
• Perform AI safety and security testing for prompt injection and data leakage.
• Establish quant metrics and gates to measure performance across releases.
• Build/maintain automated test suites across APIs, UI, integration, RAG, agents.

Conhecimentos

GenAI testing

LLM testing

Automation frameworks

Test automation

Java

Python

Formação académica

Bachelor's degree in Computer Science/Engineering/IT

Ferramentas

Langsmith

Langfuse

Ragas

Embedding evaluation tools

Descrição da oferta de emprego

Job Description: Job Title: Quality Assurance Engineer – AI Agentic Applications Location: Pune, India Corporate Title: AVP

Role Description

Quality Assurance Engineer to validate AI agentic applications across development and production. The position combines specialized AI testing, automation, safety evaluation, and continuous quality measurement. The engineer is responsible to design, build and execute the test strategy for LLM accuracy, RAG ingestion and attribution, agent planning and tool use, recovery behavior, and production failure patterns.

What we’ll offer you

• Best in class leave policy
• Gender neutral parental leaves
• 100% reimbursement under childcare assistance benefit (gender neutral)
• Sponsorship for Industry relevant certifications and education
• Employee Assistance Program for you and your family members
• Comprehensive Hospitalization Insurance for you and your dependents
• Accident and Term life Insurance
• Complementary Health screening for 35 yrs. and above

Your key responsibilities

• Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows
• Develop and implement specialized testing approaches for LLM-powered applications, validating response accuracy, relevance, and grounding, while identifying hallucinations and edge-case behaviours
• Test and evaluate the behaviour of autonomous and semi-autonomous AI agents, ensuring proper planning, tool selection/sequencing, agent-to-agent interactions, and graceful error/timeout recovery
• Validate Retrieval-Augmented Generation (RAG) performance, including document ingestion, chunking, metadata quality, embedding accuracy, and citation/source attribution
• Perform AI safety and security testing to identify vulnerabilities such as prompt injection (direct/indirect), data leakage, unauthorized tool execution, and privilege escalation
• Establish quantitative AI evaluation metrics and quality gates (e.g., classification accuracy, groundedness, hallucination rate, tool-selection accuracy, and cost/latency) to measure performance across releases
• Build and maintain automated test suites (covering APIs, UI, integration, RAG, and agent trajectories) to progressively transition the testing suite toward automated continuous evaluation
• Analyze production AI interactions and failed agent journeys to identify recurring failure patterns, create regression tests from production incidents, and ensure decision auditability

Minimum Required Skills & Experience (Must Have)

• Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline
• 5+ years of experience in software quality assurance and testing, with strong knowledge of test planning, functional, integration, regression, and end-to-end testing
• 1–2 years of hands‑on experience specifically testing GenAI, LLM, or Agentic AI applications
• Strong experience with test automation and building automation frameworks and test utilities (rather than relying solely on manual tools)
• Strong programming ability in at least one language: Java / Kotlin or Python
• Strong understanding of generative AI technologies, including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents/agentic workflows, and tool/function calling
• Hands‑on experience with LLM and agent evaluation tools such as Langsmith/Langfuse, Ragas, or equivalent frameworks for dataset‑based testing, custom evaluation metrics, experiment tracking, tracing, and continuous regression evaluation

Add On Experience (Nice to Have)

• Experience testing API and data integrations, including REST APIs, JSON, SQL, databases, and messaging systems like Kafka
• Good understanding of CI/CD integration and establishing automated quality gates
• Experience working in the financial services industry, with knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls
• Experience with AI/LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies
• Hands‑on experience with advanced AI testing capabilities such as prompt injection testing, AI evaluation/evals, and AI observability
• Experience testing distributed systems
• Proven ability to leverage AI tools to enhance productivity, optimise workflows to solve business problems, while applying critical judgment to ensure responsible and ethical use of data and AI outputs.

How we’ll support you

• Training and development to help you excel in your career
• Coaching and support from experts in your team
• A culture of continuous learning to aid progression
• A range of flexible benefits that you can tailor to suit your needs

About us and our teams

Please visit our company website for further information: https://www.db.com/company/company.html

We strive for a culture in which we are empowered to excel together every day. This includes acting responsibly, thinking commercially, taking initiative and working collaboratively. Together we share and celebrate the successes of our people. Together we are Deutsche Bank Group.

We welcome applications from all people and promote a positive, fair and inclusive work environment.

For over 150 years, our dedication to being the Global Hausbank for our clients has been driven by our people – in around 60 countries and across more than 150 nationalities. Their deep understanding, insights, expertise, and passion help our clients navigate an increasingly complex world – be it in our Corporate Bank, our Private Bank, our Investment Bank or our Asset Management (DWS) division. Together we can make a great impact for our clients at home and abroad, securing their lasting success and financial security.

More information at: Deutsche Bank Careers (db.com)