AI TEST LEAD

Vaisesika Consulting; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Vaisesika Consulting is seeking an AI Test Lead with 7+ years of software testing and test automation experience. This remote role focuses on AI/GenAI, LLM, and RAG-based applications, defining test strategies and leading testing initiatives across API, performance, security, and AI quality dimensions.

You will build automation frameworks, validate AI-driven solutions, and mentor QA engineers while collaborating with data scientists, product owners, and developers to ensure high-quality

Qualifications

• 7+ years of experience in Software Testing / QA / Test Automation.
• Hands-on AI/GenAI/LLM testing experience.
• Experience designing automated testing frameworks.

Responsibilities

• Lead end-to-end testing for AI/ML and Generative AI apps.
• Define test strategy, plans, and quality standards.
• Develop automated testing frameworks for web, API, and AI apps.
• Validate AI responses for accuracy, bias, toxicity, and hallucination.
• Integrate automated tests into CI/CD pipelines.
• Mentor QA engineers and guide testing teams.
• Collaborate with data scientists, developers, and product owners.

Skills

AI/GenAI testing

Selenium

Playwright

Cypress

Python

Java

API testing

SQL

CI/CD

Git

Jenkins

Azure DevOps

SDLC/STLC

Tools

Postman

REST Assured

OpenAI Evals

DeepEval

LangChain

Job description

AI Test Lead

Experience: 7+ Years
Location:Remote

Position Summary

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands‑on exposure to AI/ML, Generative AI, LLM, and RAG-based applications.
The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

• Lead end-to-end testing activities for AI/ML and Generative AI applications.
• Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
• Design and implement automated testing frameworks for web, API, and AI-based applications.
• Test LLM, RAG, prompt-based applications, chatbots, and AI agents.
• Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
• Develop test datasets and evaluation methodologies for AI/GenAI applications.
• Perform API, functional, regression, integration, and end-to-end testing.
• Collaborate with developers, data scientists, product owners, and business stakeholders.
• Identify and troubleshoot defects and quality issues across application and AI components.
• Integrate automated tests into CI/CD pipelines.
• Mentor QA engineers and provide technical leadership to the testing team.
• Establish quality metrics, reporting, and continuous improvement practices.
• Ensure testing standards and best practices are followed across projects.

Required Skills

• 7+ years of experience in Software Testing / QA / Test Automation.
• Hands-on experience in AI/GenAI/LLM testing.
• Strong experience with Selenium, Playwright, Cypress, or similar automation tools.
• Strong programming skills in Python or Java.
• Experience in API testing using REST, Postman, Rest Assured, or similar tools.
• Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications.
• Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality.
• Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals, or equivalent.
• Good understanding of SQL and database testing.
• Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Good to Have

• Experience testing LangChain / LangGraph applications.
• Knowledge of vector databases and RAG architectures.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Knowledge of AI security testing and responsible AI principles.
• Experience in performance and security testing.
• Previous experience leading or mentoring QA teams.