AI TEST LEAD

Vaisesika; Bengaluru; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Vaisesika is seeking an experienced AI Test Lead in Bengaluru to drive testing for AI/GenAI-enabled software. You will define quality standards, build automation frameworks, and lead testing across API, performance, security, and AI quality dimensions.

You will mentor QA engineers, collaborate with data scientists and product teams, and ensure robust testing coverage for AI-driven applications with an emphasis on accuracy, bias mitigation, and reliability.

Qualifications

• 7+ years in Software Testing / QA / Test Automation.
• Hands-on AI/GenAI/LLM testing experience.
• Strong automation tool experience (Selenium, Playwright, Cypress).
• Programming skills in Python or Java.
• Experience with API testing using REST, Postman, Rest Assured.
• Understanding of LLM, RAG, prompt engineering and AI agents.
• Exposure to AI evaluation concepts like hallucination, bias and response quality.

Responsibilities

• Lead end-to-end testing for AI/ML and Generative AI apps.
• Define test strategy, plans and quality standards for AI solutions.
• Design and implement automated testing frameworks for web, API and AI apps.
• Test LLM, RAG, prompt-based apps, chatbots and AI agents.
• Validate AI responses for accuracy, relevance and safety.
• Develop test datasets and evaluation methodologies for GenAI apps.
• Perform API, functional, regression and integration testing.
• Collaborate with developers, data scientists, product owners and stakeholders.
• Identify and troubleshoot defects across AI components and apps.
• Integrate automated tests into CI/CD pipelines.
• Mentor QA engineers and provide technical leadership.

Skills

Test leadership

Automation

AI/GenAI testing

Python/Java

API testing

SQL/database testing

Git & CI/CD

SDLC/STLC/Agile

Education

Bachelor's degree in CS/IT/Engineering

Tools

Selenium

Playwright

Cypress

Postman

Rest Assured

Git

Jenkins

GitHub Actions

Azure DevOps

Job description

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands-on exposure to AI/ML, Generative AI, LLM, and RAG-based applications.

The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

• Lead end-to-end testing activities for AI/ML and Generative AI applications.
• Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
• Design and implement automated testing frameworks for web, API, and AI-based applications.
• Test LLM, RAG, prompt-based applications, chatbots, and AI agents.
• Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
• Develop test datasets and evaluation methodologies for AI/GenAI applications.
• Perform API, functional, regression, integration, and end-to-end testing.
• Collaborate with developers, data scientists, product owners, and business stakeholders.
• Identify and troubleshoot defects and quality issues across application and AI components.
• Integrate automated tests into CI/CD pipelines.
• Mentor QA engineers and provide technical leadership to the testing team.
• Establish quality metrics, reporting, and continuous improvement practices.
• Ensure testing standards and best practices are followed across projects.

Required Skills

• 7+ years of experience in Software Testing / QA / Test Automation.
• Hands-on experience in AI/GenAI/LLM testing.
• Strong experience with Selenium, Playwright, Cypress, or similar automation tools.
• Strong programming skills in Python or Java.
• Experience in API testing using REST, Postman, Rest Assured, or similar tools.
• Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications.
• Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality.
• Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals, or equivalent.
• Good understanding of SQL and database testing.
• Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Good to Have

• Experience testing LangChain / LangGraph applications.
• Knowledge of vector databases and RAG architectures.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Knowledge of AI security testing and responsible AI principles.
• Experience in performance and security testing.
• Previous experience leading or mentoring QA teams.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Candidate Profile

The ideal candidate should have strong test leadership and automation experience along with practical exposure to AI/GenAI testing. The ability to work closely with engineering, data science, and product teams and translate AI quality requirements into effective testing strategies is essential.