AI Tester (Automation)

Hiringhood; Greater Noida; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Hiringhood in Noida is seeking an AI Tester (Automation) to design and execute testing strategies for AI/ML and LLM-powered applications. You will validate model accuracy, performance, reliability, and output quality while building automated test frameworks with Selenium, Cypress, Playwright and Appium.

Join a team of data scientists and ML engineers to develop AI evaluation pipelines, perform adversarial and edge-case testing, ensure data quality, and integrate test suites into CI/CD pipelines

Qualifications

• 2+ years in QA automation + AI testing.
• Strong in Test Automation — Selenium, Cypress, Playwright, Appium.
• Hands-on with testing AI/ML models or LLM-powered features.
• Familiar with prompt engineering & how prompts impact AI output quality.
• Proficient in API testing — Postman, RestAssured.
• Comfortable scripting in Python or JavaScript.
• Understanding of ML concepts — accuracy, precision, recall, F1 score.
• Familiar with AI testing tools — Pytest, MLflow, Great Expectations.
• Hands-on with CI/CD — Jenkins, GitHub Actions, or GitLab CI.
• Self-starter with ownership mindset & proactive communication.
• Ready to work onsite in Noida.
• Excellent communication and collaboration skills.

Responsibilities

• Design & execute AI/ML model test strategies accuracy, performance, consistency & reliability
• Test LLM-powered features for hallucinations, prompt injection, bias & output quality
• Build & maintain automated test frameworks — Selenium, Cypress, Playwright, Appium
• Develop AI evaluation pipelines — test LLM responses for relevance, coherence & toxicity
• Perform adversarial testing & edge case generation — probe AI with boundary inputs
• Validate data quality & training datasets — identify noise, bias & inconsistencies
• Conduct regression testing to ensure model updates don't degrade AI performance
• Monitor AI model drift & flag performance degradation in production
• Execute API & integration testing for AI services — Postman or RestAssured
• Integrate AI test suites into CI/CD pipelines — Jenkins, GitHub Actions
• Log & track AI-specific bugs — hallucinations, false positives, model failures
• Collaborate with Data Science & ML engineers on quality benchmarks
• Participate in agile ceremonies as the AI quality advocate

Skills

QA automation + AI testing

Selenium

Cypress

Playwright

Appium

Python

JavaScript

Postman

RestAssured

ML concepts

CI/CD

Jenkins

GitHub Actions

GitLab CI

Communication

Tools

Selenium

Cypress

Playwright

Appium

Postman

RestAssured

Pytest

MLflow

Great Expectations

Jenkins

GitHub Actions

Job description

Opportunity Description AI Tester (Automation)

Our end client — a technology solutions and digital engineering organization specializing in software development, quality engineering, automation, cloud, and emerging technologies — delivers scalable and innovative technology solutions for businesses across industries. The organization focuses on helping clients accelerate digital transformation through modern engineering practices, automation, and AI-driven solutions.

They are looking for an experienced AI Tester (Automation) to design and execute testing strategies for AI/ML and LLM-powered applications. The role involves validating model accuracy, performance, reliability, and output quality, along with building automated test frameworks using tools such as Selenium, Cypress, Playwright, and Appium. The professional will also work on AI evaluation pipelines, adversarial testing, hallucination and prompt-injection testing, edge‑case validation, and data quality to ensure robust and production‑ready AI solutions.

Roles & Responsibilities

• Design & execute AI/ML model test strategies accuracy, performance, consistency & reliability
• Test LLM-powered features for hallucinations, prompt injection, bias & output quality
• Build & maintain automated test frameworks — Selenium, Cypress, Playwright, Appium
• Develop AI evaluation pipelines — test LLM responses for relevance, coherence & toxicity
• Perform adversarial testing & edge case generation — probe AI with boundary inputs
• Validate data quality & training datasets — identify noise, bias & inconsistencies
• Conduct regression testing to ensure model updates don't degrade AI performance
• Monitor AI model drift & flag performance degradation in production
• Execute API & integration testing for AI services — Postman or RestAssured
• Integrate AI test suites into CI/CD pipelines — Jenkins, GitHub Actions
• Log & track AI-specific bugs — hallucinations, false positives, model failures
• Collaborate with Data Science & ML engineers on quality benchmarks
• Participate in agile ceremonies as the AI quality advocate

Required Skills

• A curious, detail‑obsessed tester with 2+ years in QA automation + AI testing
• Strong in Test Automation — Selenium / Cypress / Playwright / Appium Must Have
• Hands‑on with testing AI/ML models or LLM-powered features
• Familiar with prompt engineering & how prompts impact AI output quality
• Proficient in API testing — Postman, RestAssured
• Comfortable scripting in Python or JavaScript
• Understanding of ML concepts — accuracy, precision, recall, F1 score
• Familiar with AI testing tools — Pytest, MLflow, Great Expectations
• Hands‑on with CI/CD — Jenkins, GitHub Actions, or GitLab CI
• Self‑starter with ownership mindset & proactive communication
• Ready to work onsite in Noida
• Excellent communication and collaboration skills.