QA Engineer

YO IT Consulting; Mumbai; Hybrid; INR 800,000 - 1,400,000; Full time

Job summary

YO IT Consulting is seeking a QA Engineer - AI Initiatives to ensure quality and performance of AI/ML-powered products. This role involves close collaboration with Data Scientists and ML Engineers to deliver robust AI solutions while addressing challenges like bias and model drift.

Ideal candidates should have a Bachelor's or Master's degree in Computer Science and 3-6 years of QA experience, particularly in AI/ML. Excellent skills in Python and familiarity with testing tools such as Pytest and Selenium are essential.

Qualifications

• 3 - 6 years of professional QA experience, with at least 1 -2 years in AI/ML quality assurance.
• Strong proficiency in Python for test automation and data analysis.
• Hands‑on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms.
• Solid understanding of the ML lifecycle training, validation, deployment, and monitoring phases.

Responsibilities

• Design and execute comprehensive testing strategies for AI/ML models and data pipelines.
• Develop and maintain automated testing frameworks for model validation.
• Evaluate AI model outputs for accuracy, consistency, and bias.
• Monitor production AI systems for model drift and performance degradation.
• Document defects and AI‑specific failure patterns.

Skills

Expertise in AI/ML model testing and validation

Strong understanding of LLM behavior, hallucination detection, and bias evaluation

Experience testing RAG applications, chatbots, recommendation systems

Strong analytical and problem-solving abilities

Excellent documentation and communication skills

Ability to collaborate effectively with Data Science and Engineering teams

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Tools

Python

Pytest

Selenium

Postman

Job description

Job Description

Location

Mumbai / Bangalore, India

Experience Required

3-6 Years

Job Type

Full-time, Hybrid (14 Days/Month Work from Office)

Work Timings

12:30 PM - 9:30 PM

Interview Process

3 Virtual Rounds + 1 In-Person Round

Must Haves

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3 - 6 years of professional QA experience, with at least 1 -2 years in AI/ML quality assurance.
• Strong proficiency in Python for test automation and data analysis.
• Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith).
• Hands‑on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms.
• Solid understanding of the ML lifecycle training, validation, deployment, and monitoring phases.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 3-6 years of professional QA experience.
• Minimum 1-2 years of hands‑on experience in AI/ML Quality Assurance.
• Strong proficiency in Python for test automation and data analysis.
• Familiarity with LLM evaluation frameworks such as RAGAS, DeepEval, Promptfoo, and LangSmith.
• Experience with testing tools including Pytest, Selenium, Postman, or similar platforms.
• Solid understanding of the Machine Learning lifecycle, including training, validation, deployment, and monitoring.
• Knowledge of data quality tools and pipeline testing frameworks such as Great Expectations and dbt tests.

Skills Required

• Expertise in AI/ML model testing and validation.
• Strong understanding of LLM behavior, hallucination detection, bias evaluation, and model drift monitoring.
• Experience testing RAG (Retrieval‑Augmented Generation) applications, chatbots, recommendation systems, and AI‑powered solutions.
• Strong analytical and problem‑solving abilities.
• Ability to create and maintain evaluation datasets, ground truth datasets, and adversarial test cases.
• Experience with data pipeline testing and data integrity validation.
• Excellent documentation and communication skills.
• Ability to collaborate effectively with Data Science, Engineering, and Product teams.
• Soft skills: Analytical & Inquisitive Mindset, Red‑Team Thinking, Attention to Detail, Communication, Collaboration, Problem‑Solving.

Role Overview

We are looking for a skilled QA Engineer - AI Initiatives to ensure the quality, reliability, fairness, and performance of AI/ML‑powered products and systems. The ideal candidate should possess strong expertise in AI quality assurance, model evaluation, automation testing, and data validation.

This role provides an opportunity to work closely with Data Scientists, ML Engineers, and Product teams to deliver robust, ethical, and high‑performing AI solutions while addressing AI‑specific challenges such as hallucinations, bias, model drift, and output consistency.

Key Responsibilities

• Design and execute comprehensive testing strategies for AI/ML models, LLM‑based applications, and data pipelines.
• Develop and maintain automated testing frameworks for model validation, regression testing, and performance benchmarking.
• Evaluate AI model outputs for accuracy, consistency, relevance, hallucinations, and bias.
• Test RAG pipelines, chatbots, recommendation systems, and other AI‑powered features.
• Collaborate with Data Scientists and ML Engineers to define quality benchmarks and acceptance criteria.
• Build and maintain evaluation datasets, ground truth datasets, and adversarial testing scenarios.
• Monitor production AI systems for model drift, performance degradation, and anomalous behavior.
• Validate data quality, feature stores, and data pipelines feeding AI systems.
• Document defects, edge cases, and AI‑specific failure patterns with actionable recommendations.
• Conduct fairness assessments, bias audits, and explainability checks to ensure ethical AI deployment.
• Support continuous improvement of AI testing methodologies, tools, and frameworks.

Nice To Have

• Experience with prompt engineering and red‑teaming LLMs
• Familiarity with MLOps platforms such as MLflow, SageMaker, or Vertex AI
• Knowledge of vector databases and embedding quality evaluation
• Understanding of AI safety, responsible AI principles, and fairness frameworks
• Experience with A/B testing and shadow deployment strategiesKnowledge of CI/CD pipelines and DevOps practices in ML environments