Agentic AI Testing Lead

Ishareinc; Dadri; On-site; INR 4,200,000 - 6,600,000; Full time

Job summary

Ishareinc in Noida is seeking an experienced Agentic AI Testing Lead to build and head an AI quality engineering team responsible for validating LLM-based, RAG, multi-agent, and autonomous AI solutions.

The successful candidate will define testing strategies, evaluation frameworks, automation pipelines, quality metrics, and governance standards to ensure AI systems are reliable, accurate, safe, scalable, and production-ready. Immediate joiners preferred.

Qualifications

• 8–12 years of experience in software testing, quality engineering, or test automation.
• At least 3 years hands‑on GenAI, LLM testing, Agentic AI testing, or AI Quality Engineering experience.

Responsibilities

• Define testing strategies for LLM, RAG, multi-agent, and Agentic AI systems.
• Validate agent reasoning, memory, tool usage, prompt behavior, API integrations, database interactions, and end-to-end workflows.
• Evaluate outputs for accuracy, relevance, groundedness, consistency, completeness, hallucination risk, toxicity, bias, and guardrail compliance.
• Establish AI quality KPIs, including hallucination rate, groundedness score, agent success rate, task-completion rate, response relevance, latency, cost efficiency, and user satisfaction.
• Build automated evaluation pipelines, scoring mechanisms, quality dashboards, and CI/CD‑integrated quality gates.
• Develop reusable test harnesses, simulators, and benchmarking frameworks for comparing models, prompts, tools, and agent configurations.
• Conduct regression, performance, security, reliability, and adversarial testing for AI solutions.

Skills

Analytical thinking

Communication

Problem solving

Leadership

Stakeholder management

Quality engineering mindset

Tools

Python

Pytest

Playwright

API automation

CI/CD pipelines

DeepEval

Ragas

LangSmith

OpenAI Evals

Promptfoo

TruLens

Azure

AWS

GCP

Job description

Job Title: Agentic AI Testing Lead

Experience: 8 -12 years
Location: Noida
Employment Type: Full-time
Joining Preference: Immediate joiners preferred

Role Overview

We are seeking an experienced Agentic AI Testing Lead to build and lead an AI Quality Engineering team responsible for validating LLM-based, RAG, multi-agent, and autonomous AI solutions.

The successful candidate will define AI testing strategies, evaluation frameworks, automation pipelines, quality metrics, and governance standards to ensure AI systems are reliable, accurate, safe, scalable, and production-ready.

Key Responsibilities

Agentic AI Testing and Evaluation

• Define testing strategies for LLM, RAG, multi-agent, and Agentic AI systems.
• Validate agent reasoning, memory, tool usage, prompt behavior, API integrations, database interactions, and end-to-end workflows.
• Evaluate outputs for accuracy, relevance, groundedness, consistency, completeness, hallucination risk, toxicity, bias, and guardrail compliance.
• Establish AI quality KPIs, including hallucination rate, groundedness score, agent success rate, task-completion rate, response relevance, latency, cost efficiency, and user satisfaction.
• Build automated evaluation pipelines, scoring mechanisms, quality dashboards, and CI/CD-integrated quality gates.
• Develop reusable test harnesses, simulators, and benchmarking frameworks for comparing models, prompts, tools, and agent configurations.
• Conduct regression, performance, security, reliability, and adversarial testing for AI solutions.

Team Leadership

• Build and lead a team of Agentic AI Quality Engineers.
• Define team structure, testing standards, governance models, and quality-engineering best practices.
• Mentor QA engineers in AI testing methodologies, evaluation techniques, Python automation, and emerging AI testing tools.
• Collaborate with Product, Engineering, Data Science, and AI Research teams to improve AI quality.
• Drive innovation and continuous improvement across AI testing practices.

Reporting and Governance

• Present quality assessments, risk reports, KPI trends, and evaluation results to leadership and stakeholders.
• Provide recommendations for improving model, prompt, agent, and workflow performance.
• Establish quality governance for Agentic AI initiatives.
• Maintain traceability of test activities, evaluation criteria, defects, risks, and quality benchmarks.

Required Skills and Experience

• 812 years of experience in software testing, quality engineering, or test automation.
• At least 3 years of hands-on experience in GenAI, LLM testing, Agentic AI testing, or AI Quality Engineering.
• Strong understanding of LLMs, AI agents, RAG, prompt validation, tool calling, agent memory, MCP, and multi-agent orchestration.
• Experience defining AI quality metrics, evaluation methodologies, benchmarking frameworks, and model-comparison approaches.
• Strong hands‑on experience with Python, Pytest, Playwright, API automation, test‑framework development, and CI/CD quality gates.
• Experience with DeepEval, Ragas, LangSmith, OpenAI Evals, Promptfoo, TruLens, or equivalent evaluation tools.
• Exposure to Azure, AWS, or Google Cloud Platform.
• Strong analytical, communication, problem‑solving, leadership, and stakeholder‑management skills.
• Ability to lead multiple initiatives in a rapidly evolving AI environment.

Preferred Qualifications

• Experience testing enterprise Agentic AI platforms and autonomous AI systems.
• Exposure to LangGraph, CrewAI, AutoGen, Semantic Kernel, or Microsoft Copilot Studio.
• Experience with AI observability, monitoring, model governance, responsible AI, and AI safety.
• Experience developing AI quality dashboards and KPI‑reporting systems.
• ISTQB, AI Testing, GenAI, Azure AI, AWS AI, or equivalent certification.