AI QA Engineer

SIRO Clinpharm Pvt. Ltd.; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

SIRO Clinpharm Pvt. Ltd. is seeking an experienced AI/ML Quality Engineer to ensure quality, reliability, and compliance of AI/ML, GenAI, LLMs in pharma/manufacturing environments.

The ideal candidate will have expertise in model evaluation, GenAI testing, automated QA pipelines, and AI risk assessment. Regulated environments preferred.

Qualifications

• 5+ years of experience in AI/ML quality engineering
• Experience with GenAI testing, LLM evaluations
• Experience in regulated or quality-critical environments preferred
• Strong Python, APIs, scripting, and automation skills

Responsibilities

• Develop and execute AI/ML model QA strategies across the model lifecycle
• Evaluate models using Accuracy, Precision, Recall, F1, ROC/AUC, Confusion Matrix
• Test GenAI and LLM evaluations for accuracy, relevance, and hallucinations
• Design testing for AI agents and multi-agent workflows, including tool calling and output validation

Skills

Python

APIs

Scripting

Automation

LangChain

LangGraph

Langfuse

Tools

Python

APIs

Scripting

Automation frameworks

LangChain

LangGraph

Langfuse

Job description

Greetings !

We are hiring for TCS Client!

Role : AI/ML & GenAI Quality Engineer

Experience: 5+ Years

Work Mode: Remote

Role: AI/ML Model QA / GenAI Validation Engineer

Job Summary

We are looking for an experienced AI/ML & GenAI Quality Engineer to ensure the quality, reliability, performance, and compliance of AI/ML, Generative AI, Large Language Models (LLMs), and Agentic AI solutions within a pharmaceutical and manufacturing environment.

The ideal candidate will have strong expertise in AI/ML model evaluation, GenAI testing, automated quality engineering, and AI risk assessment. Experience in regulated or quality-critical environments is preferred.

Key Responsibilities

• Develop and execute AI/ML model quality assurance and evaluation strategies across the model lifecycle.
• Evaluate machine learning models using Accuracy, Precision, Recall, F1 Score, ROC/AUC, Confusion Matrix, and other relevant metrics.
• Test models across diverse datasets, edge cases, failure scenarios, and operating conditions.
• Perform GenAI and LLM evaluations for accuracy, relevance, consistency, hallucination, groundedness, and response quality.
• Design and execute testing strategies for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
• Use Python, APIs, scripting, and automation frameworks for AI testing and evaluation.
• Work with LangChain, LangGraph, Langfuse, and similar tools for agent development, tracing, observability, and evaluation.
• Identify model bias, data drift, hallucinations, instability, and AI application failures, and perform root-cause analysis.
• Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and business stakeholders.
• Document test results, model limitations, risks, defects, and evaluation outcomes in an audit-ready format.
• Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices.
• Conduct AI Red-Teaming and adversarial testing to identify vulnerabilities, unsafe outputs, and potential failure modes.