AI Quality Engineer, Safety and RAG - Assistant Vice President

Citi; Chennai District; Presencial; INR 14.409.000 - 20.173.000; Jornada completa

Descripción de la vacante

Citi seeks a seasoned AI Quality Engineer at AVP level to join the Retail and Wealth Risk Engineering team. You will own end-to-end testing for agentic AI pipelines, RAG workflows, and safety controls, ensuring reliability and production readiness at scale.

Design test strategies, validate tool-use and memory performance, and uphold privacy, security and compliance across multi-agent systems and CI/CD pipelines.

Responsabilidades

• Agentic AI Testing: design end-to-end test strategies for single-agent and multi-agent workflows; validate agent reasoning and tool-use correctness; evaluate memory systems; verify multi-agent handoffs; ensure termination and loop prevention.
• RAG Testing: design end-to-end tests for ingestion, chunking, embedding, retrieval and generation; validate retrieval relevance and vector quality; assess faithfulness and citation accuracy; test chunking strategies and context window management; ensure end-to-end regression tests.
• Test Automation: build automated test harnesses for Agentic and RAG systems; integrate into CI/CD pipelines; create data validation and quality frameworks; develop prompt regression suites; ensure determinism and reproducibility.
• AI Safety & Security Testing: perform red-teaming for jailbreaks and prompt injection; test guardrails and access controls; simulate data poisoning and backdoor attacks; evaluate bias and privacy controls.

Descripción del empleo

AI Quality Engineer, Safety and RAG - Assistant Vice President

Discover your future at Citi

Working at Citi is far more than just a job. A career with us means joining a team of approximately 219,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.

Job Overview

We are seeking a highly motivated and experienced AI Quality Engineer to join our Retail and Wealth Risk Engineering team under the Enterprise Risk Technology platform. This role spans the full spectrum of modern AI quality engineering — from Agentic AI flow testing and RAG pipeline validation to AI safety, test automation, and performance & reliability engineering.

You will be the quality pillar for complex autonomous AI systems, ensuring they are safe, accurate, explainable, resilient, and production-ready at scale. This is a high-impact, highly technical role that requires both depth in AI/ML and breadth across testing disciplines.

Responsibilities

Agentic AI Testing

• Design and execute end-to-end test strategies for Agentic AI pipelines, including single-agent and multi-agent workflows.
• Validate agent reasoning, planning, and decision-making chains (e.g., ReAct, Chain-of-Thought, Plan-and-Execute, Reflexion).
• Test tool-use correctness — ensuring agents invoke the right tools, with correct parameters, at the right time.
• Evaluate agent memory systems (short-term, long-term, episodic) for accuracy and context retention across sessions.
• Validate agent handoff and delegation logic in multi-agent orchestration frameworks (e.g., AutoGen, CrewAI, LangGraph).
• Test termination conditions, loop detection, and infinite loop prevention in autonomous agent loops.

RAG (Retrieval-Augmented Generation) Testing

• Design comprehensive test strategies for end-to-end RAG pipelines — covering ingestion, chunking, embedding, retrieval, reranking, and generation stages.
• Validate retrieval accuracy and relevance — ensuring the correct context chunks are retrieved for a given query.
• Test embedding model quality and vector similarity thresholds across different document corpora.
• Evaluate faithfulness, groundedness, and answer relevance of generated responses using frameworks like RAGAS, TruLens, DeepEval.
• Test chunking strategies (fixed, semantic, hierarchical) for their impact on retrieval quality.
• Validate context window management — ensuring retrieved context does not exceed token limits or degrade generation quality.
• Conduct end-to-end regression testing when the underlying knowledge base, embedding model, or LLM changes.
• Test multi-turn conversational RAG for context coherence and citation accuracy across turns.

Test Automation

• Build and maintain automated test harnesses for Agentic and RAG systems, including agent trajectory replay, tool mock injection, and prompt simulation.
• Develop automated evaluation pipelines integrated into CI/CD workflows for continuous model and agent validation.
• Create data validation and data quality frameworks (using Great Expectations, Deequ, or custom tooling) for training, retrieval, and inference data.
• Build prompt regression suites to detect behavioral drift across LLM versions or prompt changes.
• Implement determinism and reproducibility tests for stochastic LLM-based decisions.
• Automate vector database validation — index integrity, embedding drift, and retrieval consistency checks.

AI Safety & Security Testing

• Conduct red-teaming and adversarial testing to uncover jailbreaks, prompt injection vulnerabilities, and goal misalignment in LLM-based systems.
• Test output guardrails and content filters for unsafe, biased, toxic, or out-of-scope model behavior.
• Validate privilege escalation controls — ensuring agents do not exceed permitted actions or access unauthorized resources.
• Perform data poisoning and backdoor attack simulations to assess model robustness.
• Evaluate models for bias, fairness, and discrimination using frameworks such as AI Fairness 360 and Aequitas.
• Test PII leakage and data privacy controls in RAG and agent pipelines in accordance with GDPR,