Test Team Lead

QualiZeal; Hyderabad; On-site; INR 4,000,000 - 8,500,000; Full time

Job summary

QualiZeal is seeking an experienced AI Test Lead to drive end-to-end testing and quality assurance for Generative AI applications. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers and DevOps to validate production-ready AI solutions against industry testing standards.

You will lead AI test strategy, plan resources, manage functional/non-functional testing, mentor QA engineers, and ensure delivery timelines align with agile cycles.

Qualifications

• 8+ years of software testing and quality assurance experience.
• 1–2 years hands-on AI/ML and GenAI testing experience.
• Proven experience managing end-to-end testing activities and QA teams.
• Track record of testing production-grade AI applications through complete release cycles.

Responsibilities

• Lead end-to-end AI testing activities, including strategy, planning, design, execution, defect management, and sign-off.
• Define and implement test strategies for LLM-powered applications including RAG and AI agents.
• Plan functional and non-functional testing: model evaluation, prompt regression, bias/fairness, robustness, hallucination detection, explainability validation.
• Track test coverage, metrics, and release readiness; report to QA leadership.
• Act as primary testing contact for customers; lead requirement discussions and defect triage calls.
• Translate customer requirements into comprehensive AI test plans and scenarios.
• Manage day-to-day QA team activities: task allocation, monitoring, risk, escalation.
• Present results and dashboards to customer stakeholders in business language.
• Drive UAT support, release sign-off, and post-release quality monitoring.

Skills

AI Testing

Python scripting

Pytest

Selenium

Playwright

API testing

CI/CD

Customer engagement

Test automation

Leadership

Education

Bachelor's or Master's in CS/Engineering/AI/ML

ISTQB CT-AI or equivalent

Tools

Jira

Zephyr

TestRail

Job description

We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.

Key Responsibilities

AI Testing Strategy & Delivery Management

• Lead full lifecycle of AI testing activities: test strategy, planning, design, execution, defect management, and sign-off for GenAI applications
• Define and implement test strategies for LLM-powered applications including RAG systems, conversational AI, and AI agents
• Plan and manage functional, non-functional, and AI-specific testing: model evaluation, prompt regression, bias/fairness, robustness, hallucination detection, and explainability validation
• Establish test estimation, sprint-level test planning, milestones, and delivery timelines aligned with agile development cycles
• Track and report test coverage, quality metrics, and release readiness to QA leadership and stakeholders
• Act as the primary testing point of contact for customers; lead requirement discussions, status reviews, and defect triage calls
• Translate customer business requirements and acceptance criteria into comprehensive AI test plans and test scenarios
• Manage day-to-day testing activities of the QA team: task allocation, progress monitoring, risk identification, and escalation management
• Present test results, quality dashboards, and risk assessments to customer stakeholders in business-focused language
• Drive UAT support, release sign-off, and post-release quality monitoring in coordination with customer teams

AI Testing Standards & Quality Governance

• Implement AI testing practices aligned with industry standards: ISO/IEC 42001 (AI management systems)
• Apply responsible AI and risk frameworks such as NIST AI RMF and EU AI Act requirements to test planning and evidence collection
• Define quality gates, entry/exit criteria, and audit-ready test documentation for AI systems
• Establish evaluation benchmarks and metrics for LLM outputs: accuracy, relevance, groundedness, toxicity, and consistency

Test Automation & Tooling

• Design and build test automation frameworks primarily using Python (Pytest, Selenium, Playwright, Requests)
• Automate AI/LLM evaluation pipelines using frameworks such as DeepEval, Ragas, LangSmith, or promptfoo
• Integrate automated test suites into CI/CD pipelines (GitHub Actions, Jenkins) with reporting and alerting
• Drive API, integration, and performance testing for AI services and microservices
• Mentor QA engineers on Python scripting, automation best practices, and AI testing techniques

Required Skills & Qualifications

Experience

• 8+ years of overall experience in software testing and quality assurance
• 1–2 years of hands‑on experience in AI/ML and GenAI testing activities
• Proven experience interacting with customers and managing end-to-end testing activities and QA teams
• Track record of testing production‑grade AI applications through complete release cycles

Technical Knowledge

• Strong understanding of LLMs (GPT, Claude, Llama), prompt engineering, RAG architectures, and AI agents from a testing perspective
• Hands‑on expertise in AI‑specific testing: model evaluation, bias/fairness testing, adversarial/robustness testing, hallucination detection, and prompt regression testing
• Strong automation skills with Python scripting; experience with Pytest, Selenium/Playwright, and API testing tools (Postman, Requests)
• Familiarity with LLM evaluation tools and frameworks: DeepEval, Ragas, LangSmith, Hugging Face evaluate, or equivalent
• Working knowledge of cloud platforms (AWS, Azure, GCP), CI/CD pipelines, and test management tools (Jira, Zephyr, TestRail)

AI Testing Standards

• Knowledge of AI testing and quality standards: ISO/IEC 42001, and ISTQB CT‑AI syllabus
• Understanding of responsible AI frameworks and regulations: NIST AI RMF, EU AI Act, and data privacy considerations in AI testing

Education

• Bachelor's or Master's degree in Computer Science, Engineering, AI/ML, Data Science, or related field
• ISTQB Certified Tester – AI Testing (CT‑AI) or equivalent AI/QA certifications are a plus

What This Role Offers

• Lead cutting‑edge GenAI testing programs with high visibility to customers and senior leadership
• Ownership of AI quality strategy from test planning through production sign‑off
• Work with talented AI architects, engineers, and domain experts
• Continuous learning and growth in the rapidly evolving AI testing and quality engineering space