AI Testing Engineer

Black Turtle; Bengaluru, Pune District; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

Black Turtle is seeking a Test Lead to drive QA strategy and governance for an enterprise AI platform, overseeing a team of testers and setting quality standards for LLM/agent behavior. The role focuses on safety, compliance, and the reliable operation of multi-agent systems and RAG pipelines.

The ideal candidate will lead AI/ML testing initiatives, manage test programs, and ensure that AI solutions are robust, ethical, and production-ready, working closely with data scientists, ML engineers,

Qualifications

• 9-12 years of software testing and quality engineering experience, with at least 3-4 years in AI testing.
• Proven leadership in managing test teams and delivery programs.
• Strong expertise in Python for automation and AI testing scripts.
• Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).
• Experience in CI/CD pipelines and DevOps practices.
• Strong understanding of model evaluation metrics, drift detection, and monitoring.
• Exposure to AI ethics and compliance testing.
• Knowledge of cloud-native architectures (AWS, Azure, GCP).
• Familiarity with containerization & orchestration (Docker, Kubernetes).

Responsibilities

• Lead QA strategy, governance, and delivery for an enterprise AI platform.
• Define quality standards for LLM/agent behavior and oversee safety/compliance.
• Oversee production-ready multi-agent systems and RAG pipelines.
• Lead AI/ML testing initiatives, manage test teams, ensure quality.
• Collaborate with data scientists, ML engineers, and DevOps for trustworthy solutions.

Skills

QA Automation

AI Testing

Python Scripting

Test Leadership

CI/CD

MLOps

AI Ethics

Tools

Langchain

Langgraph

CrewAI

Docker

Kubernetes

AWS

Azure

GCP

Job description

Skills:

Strong in QA/Automation Testing concepts

Worked on AI related projects (Generative AI/RAG/LLM)

Location: Bangalore/Pune
Experience: 7 to 10 Years
Onboarding Date: 0-30 Days ONLY

Role overview:

• Test Lead will drive QA strategy, governance, and delivery for an enterprise Agentic AI Platform supporting insurance workflows. This includes leading a team of testers, defining quality standards for LLM/agent behavior, overseeing safety and compliance, and ensuring

predictable, reliable operation of multi agent systems and RAG pipelines.

• Experience in leading AI/ML testing initiatives, managing test teams, ensuring quality and enterprise AI deployments. The Test Lead will be responsible for strategic planning, governance, stakeholder management, and delivery oversight, ensuring that AI solutions are

robust, ethical, and production ready.

• Experience in software testing, specializing in Artificial Intelligence (AI) systems and MLOps pipelines. The role involves validating AI/ML models, ensuring robustness of end-to-end machine learning workflows, and driving quality engineering practices across data, models, and deployment environments. The engineer will collaborate with data scientists, ML engineers, and DevOps teams to ensure trustworthy, scalable, and production-ready AI solutions.
• Strong understanding of both manual and automated testing methodologies, with the ability to design, develop, and execute test scripts.
• Responsible for the testing and delivery of parts of a product, in accordance with the customers requirements and interpersonal quality norms.
• Strong analytical and problem-solving skills, along with a proactive approach to identify and resolve issues, are essential.
• Responsible for ensuring that the testing process is efficient, effective, and aligned with project goals and timelines.

Requirement:

• 9-12 years of experience in software testing and quality engineering, with at least 3-4 years in AI testing.
• Proven leadership in managing test teams and delivery programs.
• Strong expertise in Python for automation and AI testing scripts.
• Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).).
• Experience in CI/CD pipelines and DevOps practices.
• Strong understanding of model evaluation metrics, drift detection, and monitoring.
• Exposure to AI ethics and compliance testing.
• Knowledge of cloud-native architectures (AWS, Azure, GCP).
• Familiarity with containerization & orchestration (Docker, Kubernetes).