GenAI Test Lead

QualiZeal; Hyderabad; On-site; INR 3,500,000 - 6,500,000; Full time

Benefits offered by this job

Flexible work hours

GenAI project exposure

Job summary

QualiZeal in Hyderabad invites an experienced AI Test Lead to steer end-to-end testing and QA for Generative AI apps. You will coordinate with QA Managers, AI/ML engineers, prompt engineers, full-stack developers, and DevOps to ensure production-ready AI solutions meet industry testing standards.

You will define AI testing strategies, lead multiple testing domains, and mentor QA engineers while collaborating closely with customers during requirement discussions, triage, and UAT support.

Qualifications

• 8–15+ years in software testing and QA with leadership experience.
• Hands-on AI/GenAI testing and LLM validation experience.
• Strong Python programming and test automation skills.
• Experience with API testing, automation frameworks, and CI/CD.

Responsibilities

• Define and execute AI testing strategy, planning, execution, and release sign‑off.
• Lead functional, automation, API, integration, performance, and GenAI testing.
• Design test approaches for LLM evaluation, prompt regression, and bias checks.
• Manage sprint planning, estimates, quality metrics, and release readiness.
• Serve as primary QA contact for customers, drive requirement discussions and UAT support.
• Build Python-based automation frameworks using Pytest, Playwright, and Selenium.

Skills

QA Leadership

GenAI Testing

Python

API Testing

CI/CD

Agile/Scrum

Customer Communication

Tools

Pytest

Playwright

Selenium

API automation

Job description

We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.

Key Responsibilities

• Define and execute AI testing strategy, planning, execution, defect management, and release sign-off.
• Lead functional, automation, API, integration, performance, and AI-specific testing for GenAI applications.
• Design test approaches for LLM evaluation, prompt regression, hallucination detection, bias and fairness testing, robustness validation, and output quality assessment.
• Manage sprint-level test planning, estimation, quality metrics, and release readiness in Agile environments.
• Act as the primary QA contact for customers, driving requirement discussions, status reviews, defect triage, and UAT support.
• Build and enhance Python-based automation frameworks using Pytest, Playwright, Selenium, and API automation tools.
• Implement automated AI evaluation pipelines using tools such as DeepEval, Ragas, LangSmith, or promptfoo.
• Integrate test automation into CI/CD pipelines and establish quality governance, documentation, and reporting standards.
• Mentor QA engineers and promote best practices in AI testing and automation.

Required skills

• 8-15+ years in software testing and QA, with experience leading testing teams.
• Hands-on experience in AI/GenAI testing, LLM validation, or intelligent application testing.
• Strong Python programming and test automation expertise.
• Experience with API testing, automation frameworks, and CI/CD pipelines.
• Strong customer communication, stakeholder management, and delivery leadership skills.
• Experience with Agile/Scrum methodologies and test management tools.

Preferred qualifications

• Experience with RAG systems, conversational AI, AI agents, or LLM-based applications.
• Familiarity with AI governance frameworks such as ISO/IEC 42001, NIST AI RMF, or EU AI Act.
• Knowledge of AI quality metrics including accuracy, relevance, groundedness, consistency, and toxicity evaluation.

What This Role Offers

• Lead cutting-edge GenAI testing programs with high visibility to customers and senior leadership
• Ownership of AI quality strategy from test planning through production sign-off
• Work with talented AI architects, engineers, and domain experts
• Continuous learning and growth in the rapidly evolving AI testing and quality engineering space