AI QA Engineer Consultant

Acenet; Pune District; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Acenet is seeking an AI QA Engineer/Consultant to ensure quality, reliability and compliance of AI/ML, GenAI, LLM and Agentic AI solutions. The role focuses on testing governance, production-readiness and auditable outcomes within pharma/life sciences contexts.

The candidate combines AI validation expertise with quality engineering, enabling safe and auditable AI deployments in regulated environments. Collaboration with data scientists, ML engineers and product owners is essential.

Qualifications

• Experience in AI/ML quality engineering, validation, testing and governance in regulated environments.
• Ability to design and execute AI model QA strategies across development to production lifecycles.
• Familiarity with GenAI, LLM evaluation, and multi-agent systems.

Responsibilities

• Develop and execute AI/ML model QA strategies across the model lifecycle.
• Evaluate ML models using metrics like accuracy, precision, recall, F1, ROC/AUC, and thresholds.
• Test models on diverse datasets, edge cases, failure scenarios and varying conditions.
• Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination and groundedness.
• Design testing for AI agents and multi-agent workflows including tool calling and validation.
• Build test datasets, evaluation frameworks, and automated QA pipelines for AI systems.
• Use Python, APIs, scripting and automation to test at scale.
• Work with LangChain/LangGraph/Langfuse to monitor agent behavior.
• Identify bias, drift, hallucination, instability; perform root-cause analysis and remediation planning.
• Define test cases, user stories, acceptance criteria and production-readiness criteria with stakeholders.
• Collaborate with Data Scientists, ML Engineers, Software Engineers and Product Owners to align on quality objectives.
• Document test results and risks in audit-ready formats for regulatory review.
• Support Responsible AI, AI Governance, Data Integrity and Quality-by-Design practices.

Skills

AI QA Strategy

GenAI Testing

LLM Evaluation

Agentic AI Testing

Python Automation

API Automation

LangChain / LangGraph / Langfuse

Prompt Engineering

Bias & Robustness Testing

Test Case Design

Tools

Python

Job description

Job Summary

AI QA Engineer / AI QA Consultant

Role Overview

We are seeking an AI QA Engineer / AI QA Consultant responsible for ensuring the quality, reliability, performance, robustness, and compliance of AI/ML, Generative AI, LLM, and Agentic AI solutions. This role operates within pharmaceutical/life sciences and manufacturing contexts, aligning AI testing, governance, and quality practices with regulatory expectations and production-readiness requirements. The ideal candidate blends deep AI model validation expertise with practical quality engineering discipline to enable safe, auditable AI deployments.

Key Responsibilities

• Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle, from development through production.
• Evaluate ML models using metrics such as accuracy, precision, recall, F1 score, ROC/AUC, confusion matrix, and domain-specific thresholds.
• Test models across diverse datasets, edge cases, failure scenarios, and varying operating conditions to ensure robust performance.
• Perform GenAI/LLM evaluation covering accuracy, relevance, consistency, hallucination, groundedness, and response quality.
• Design and execute testing for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines tailored to AI systems.
• Utilize Python, APIs, scripting, and automation to conduct AI testing and evaluation at scale.
• Work with LangChain, LangGraph, Langfuse, and similar AI observability and evaluation tools to monitor and validate agent behavior.
• Identify bias, model drift, hallucination, instability, and AI application/model failures, followed by root-cause analysis and remediation planning.
• Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria in collaboration with stakeholders.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and business stakeholders to align on quality objectives.
• Document test results, model limitations, risks, defects, and evaluation outcomes in audit-ready formats suitable for regulatory review.
• Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices within project governance.

Must-Have Skills

• Machine Learning Model Evaluation
• GenAI / LLM Evaluation
• Agentic AI Multi-Agent Systems
• Python, APIs Automation
• LangChain, LangGraph, Langfuse
• Prompt Engineering LLM Testing
• AI Risk, Bias Robustness Testing
• Model / AI Application Regression Testing
• AI Test Strategy Test Case Design

Good to Have

• Pharmaceutical / Life Sciences experience
• GxP / CSV / Data Integrity knowledge
• Azure / AWS / GCP
• MLOps / LLMOps / ModelOps
• RAG and Vector Databases
• Responsible AI AI Governance
• CI/CD and automated testing
• AI Red-Teaming / Adversarial Testing Experience
• 5+ years of experience in AI/ML, GenAI, Model QA/Validation, Software QA, Data Science, or Quality Engineering.
• Experience in regulated, pharmaceutical, life sciences, manufacturing, or other quality-critical environments is preferred (not mandatory).

Niche Positioning

This role targets professionals specialized in AI Quality Engineering, AI Model Validation, GenAI Testing, LLM Evaluation, and Agentic AI Testing. It differentiates from traditional software QA by emphasizing AI/ML-driven quality assurance, governance, and production readiness. The candidate synthesizes AI/ML expertise with quality engineering disciplines to deliver auditable, trustworthy AI systems in regulated contexts.