Senior Test Engineer - AI/ML

Quantiphi; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Quantiphi is seeking a Senior Test Engineer - AI/ML to design and execute automation for UI, API, and AI-enabled applications. You will own end-to-end testing, collaborate with ML and software engineers, and contribute to AI quality validation across models and prompts.

You will utilize Playwright for UI automation, REST Assured or Python Requests for API tests, and SQL for data verification, while integrating tests into CI/CD pipelines in a hybrid Mumbai/Bangalore setting.

Qualifications

• Strong knowledge of STLC, SDLC, Agile, and Scrum.
• Experience designing test strategies, test scenarios, test cases, and test execution.
• Hands-on UI automation using Playwright and API automation using REST Assured (Java) or Python Requests.
• Hands-on experience with Postman for API testing and validation.
• Strong SQL skills with database testing and backend data validation.
• Experience integrating automation suites with CI/CD pipelines using Jenkins, GitHub Actions, or similar.
• Proficiency with Git and version control best practices.
• Experience testing AI/ML or Generative AI applications.
• Understanding of LLMs, Retrieval-Augmented Generation (RAG), chatbots, copilots, or AI agents.
• Experience validating AI responses for accuracy, relevance, consistency, hallucinations, toxicity, bias, and safety.
• Experience designing test scenarios for prompt engineering and AI workflow validation.
• Understanding of AI evaluation techniques and automated output validation.
• Familiarity with AI evaluation frameworks or libraries (e.g., DeepEval, LangSmith, Promptfoo).
• Experience using GenAI tools to accelerate test design and automation.

Responsibilities

• Design, develop, and maintain UI and API automation test suites using modern automation frameworks.
• Own end-to-end testing activities including test planning, automation development, execution, defect management, and reporting.
• Validate AI/ML applications, Generative AI solutions, LLM-powered apps, and intelligent workflows.
• Design and execute functional, integration, regression, and end-to-end test automation for traditional and AI-enabled applications.
• Develop and maintain reusable automation frameworks, libraries, and utilities.
• Automate API testing using REST Assured (Java) or Python Requests.
• Develop UI automation using Playwright.
• Validate backend data using SQL and perform database verification.
• Develop automated validations for AI model outputs, prompt responses, RAG pipelines, and chatbot conversations.
• Validate AI responses for correctness, consistency, relevance, grounding, safety, hallucinations, and bias.
• Collaborate with ML Engineers, Software Engineers, and Product Owners to define AI quality validation strategies.
• Integrate automated test suites into CI/CD pipelines and implement quality gates.
• Leverage GenAI tools to accelerate test case generation, automation development, debugging, documentation, and productivity.
• Contribute to AI testing frameworks, accelerators, and Quality Engineering best practices.
• Mentor junior engineers and promote automation-first and AI-first Quality Engineering practices.

Skills

UI automation

API automation

Playwright

REST Assured

Java

Python

SQL

CI/CD

AI testing

LLMs

Tools

Git

Postman

Jenkins

GitHub Actions

Kubernetes

Docker

Job description

While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Role: Senior Test Engineer - AI/ML

Experience: 3-6 years

Location: Mumbai/Bangalore (Hybrid)

Key Responsibilities:

• Design, develop, and maintain UI and API automation test suites using modern automation frameworks.


• Independently own end-to-end testing activities including test planning, automation development, execution, defect management, and reporting.


• Validate AI/ML applications, Generative AI solutions, LLM-powered applications, and intelligent workflows.


• Design and execute functional, integration, regression, and end-to-end test automation for traditional and AI-enabled applications.


• Develop and maintain reusable automation frameworks, libraries, and utilities.


• Automate API testing using REST Assured (Java) or Python Requests.


• Develop UI automation using Playwright.


• Validate backend data using SQL and perform database verification.


• Develop automated validations for AI model outputs, prompt responses, RAG pipelines, and chatbot conversations.


• Validate AI responses for correctness, consistency, relevance, grounding, safety, hallucinations, and bias.


• Collaborate with ML Engineers, Software Engineers, and Product Owners to define AI quality validation strategies.


• Integrate automated test suites into CI/CD pipelines and implement quality gates.


• Leverage GenAI tools to accelerate test case generation, automation development, debugging, documentation, and productivity.


• Contribute to AI testing frameworks, accelerators, and Quality Engineering best practices.


• Mentor junior engineers and promote automation-first and AI-first Quality Engineering practices.

Must have skills:

• Strong understanding of Software Testing Life Cycle (STLC), SDLC, Agile, and Scrum methodologies.


• Hands-on experience in designing test strategies, test scenarios, test cases, and test execution.


• Strong expertise in UI automation using Playwright and API automation using REST Assured (Java) or Python Requests.


• Hands-on experience with Postman for API testing and validation.


• Strong SQL skills with hands-on experience in database testing and backend data validation.


• Experience integrating automation suites with CI/CD pipelines using Jenkins, GitHub Actions, or similar.


• Proficiency with Git and version control best practices.


• Hands-on experience testing AI/ML or Generative AI applications.


• Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), chatbots, copilots, or AI agents.


• Experience validating AI responses for accuracy, relevance, consistency, hallucinations, toxicity, bias, and safety.


• Experience designing test scenarios for prompt engineering and AI workflow validation.


• Understanding of AI evaluation techniques and automated output validation.


• Familiarity with AI evaluation frameworks or libraries (e.g., DeepEval, Ragas, LangSmith, Promptfoo, MLflow, or similar).


• Experience using GenAI tools such as ChatGPT, GitHub Copilot, Kiro, Claude, or similar to accelerate test design, automation development, debugging, test data generation, and documentation.


• Strong debugging, root cause analysis, and analytical problem-solving skills.


• Ability to independently own end-to-end testing activities with minimal supervision.


• Excellent communication, collaboration, and stakeholder management skills.


• Ability to mentor junior engineers and promote automation and Quality Engineering best practices.


• Strong attention to detail with a quality-first and automation-first mindset.

Nice to have skills

• Knowledge of AWS services such as Amazon Bedrock, SageMaker, EC2, Lambda, API Gateway, S3, RDS, DynamoDB, CloudWatch, ECS/EKS, and IAM.


• Experience testing cloud-native applications deployed on AWS.


• Experience in Data Warehouse, ETL, or Data Pipeline testing.


• Experience with performance testing tools such as JMeter or k6.


• Exposure to Docker, Kubernetes, and containerized application testing.


• Relevant certifications in Playwright, AWS, Quality Engineering, or Agile are an added advantage.

If you like wild growth and working with happy, enthusiastic over-achievers, you’ll enjoy your career with us!