AI Quality Engineer, Safety and RAG – Assistant Vice President

Growth For Impact; Chennai District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Growth For Impact is seeking a QA Engineer to design and execute end-to-end test strategies for Agentic AI pipelines, including multi-agent workflows. You will validate tool calls, memory retention, and context management to ensure robust performance across sessions.

The role involves testing retrieval, embedding quality, accuracy of responses, and safeguarding against jailbreaks or misalignment, with CI/CD test harnesses and regression suites to support rapid, reliable AI development.

Qualifications

• Design and implement robust end-to-end test strategies for AI pipelines and workflows.
• Assess tool-use correctness, ensuring calls and parameters are accurate across steps.
• Evaluate memory and context retention in agent sessions.

Responsibilities

• Design and execute end-to-end test strategies for Agentic AI pipelines, including single‑agent and multi‑agent workflows.
• Test tool‑use correctness, ensuring agents invoke the right tools with correct parameters at the right time.
• Evaluate agent memory systems for accuracy and context retention across sessions.
• Test termination conditions, loop detection, and infinite‑loop prevention in autonomous agent loops.
• Validate retrieval accuracy and relevance for queries, ensuring correct context chunks are used.
• Test embedding model quality and vector similarity thresholds across corpora.
• Evaluate faithfulness and answer relevance using RAG frameworks like TruLens and DeepEval.
• Validate context-window management to prevent token limit issues.

Skills

Attention to detail

Analytical thinking

Tools

CI/CD pipelines

Vector databases

Security testing tools

Job description

ROLE

• Design and execute end-to-end test strategies for Agentic AI pipelines, including single‑agent and multi‑agent workflows.
• Test tool‑use correctness, ensuring agents invoke the right tools with correct parameters at the right time.
• Evaluate agent memory systems for accuracy and context retention across sessions.
• Test termination conditions, loop detection, and infinite‑loop prevention in autonomous agent loops.
• Validate retrieval accuracy and relevance, ensuring correct context chunks are retrieved for queries.
• Test embedding model quality and vector similarity thresholds across document corpora.
• Evaluate faithfulness, groundedness, and answer relevance of generated responses using frameworks such as RAGAS, TruLens, DeepEval.
• Validate context‑window management to prevent token limit exceedance or degradation of generation quality.
• Conduct regression testing when knowledge bases, embedding models, or LLMs change.
• Test multi‑turn conversational RAG for context coherence and citation accuracy across turns.
• Build and maintain automated test harnesses for Agentic and RAG systems, including agent trajectory replay, tool mock injection, and prompt simulation.
• Develop automated evaluation pipelines integrated into CI/CD workflows for continuous model and agent validation.
• Build prompt regression suites to detect behavioral drift across LLM versions or prompt changes.
• Implement determinism and reproducibility tests for stochastic LLM‑based decisions.
• Automate vector‑database validation, including index integrity, embedding drift, and retrieval consistency checks.
• Conduct red‑team and adversarial testing to uncover jailbreaks, prompt‑injection vulnerabilities, and goal misalignment in LLM‑based systems.
• Test output guardrails and content filters for unsafe, biased, toxic, or out‑of‑scope model behavior.
• Validate privilege‑escalation controls to ensure agents do not exceed permitted actions or access unauthorized resources.
• Perform data‑poisoning and backdoor attack simulations to assess model robustness.
• Evaluate models for bias, fairness, and discrimination using frameworks such as AI Fairness 360 and Aequitas.
• Test PII leakage and data‑privacy controls in RAG and agent pipelines in accordance with GDPR, CCPA, and internal policies.
• Conduct security testing aligned with OWASP Top10 for LLM applications, including prompt injection, insecure output handling, training data poisoning, insecure plugin/tool design, and sensitive information disclosure.
• Validate constitutional AI constraints, RLHF‑aligned behavior boundaries, and system‑prompt integrity.
• Collaborate with cybersecurity teams on AI‑specific threat modeling and vulnerability management; maintain safety testing playbooks and document red‑team findings.
• Define and execute load, stress, soak, and spike testing for AI‑powered APIs, inference endpoints, and agent orchestration services.
• Measure and optimize end‑to‑end latency across RAG and agentic pipelines; benchmark LLM inference throughput and identify bottlenecks.
• Test auto‑scaling behavior of AI services under variable load; validate circuit‑breaker, retry, and fallback mechanisms for graceful degradation.
• Conduct cost‑efficiency analysis of token consumption, API call costs, and infrastructure spend per agent task.
• Establish