Senior AI QA Engineer - GenAI & LLM Evaluator (Remote)

SIRO Clinpharm Pvt. Ltd.; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

SIRO Clinpharm Pvt. Ltd. is seeking an AI QA Engineer/Consultant to ensure quality, reliability, and compliance of AI/ML and GenAI solutions within Pfizer's pharmaceutical environment.

The role emphasizes model evaluation, test strategy, and automation, collaborating with data scientists, ML engineers, and product stakeholders. The ideal candidate has 5+ years in AI/ML QA, strong Python scripting, and experience with LangChain/LangGraph.

Qualifications

• Must have 5+ years in AI/ML, GenAI, model QA/validation, software QA, data science, or quality engineering.
• Experience preferred in regulated or quality-critical environments.

Responsibilities

• Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle.
• Evaluate ML models using metrics like accuracy, precision, recall, F1, ROC/AUC.
• Test models across datasets, edge cases, failure scenarios, and operating conditions.
• Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination, groundedness.
• Design testing for AI agents and multi-agent workflows, including orchestration and tool calling.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
• Use Python, APIs, scripting, and automation for AI testing and evaluation.
• Work with LangChain, LangGraph, Langfuse for agent development and observability.
• Identify bias, drift, hallucination, instability; perform root-cause analysis.
• Define test cases, user stories, acceptance criteria, and production-readiness criteria.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, product owners.
• Document results and risks in audit-ready format; support Responsible AI initiatives.

Skills

ML Model Evaluation

GenAI / LLM Evaluation

Agentic AI

Test Strategy

Python Automation

LangChain / LangGraph

Prompt Engineering

AI Risk & Bias

Model Regression Testing

Stakeholder Management

Job description

SIRO Clinpharm Pvt. Ltd. is seeking an AI QA Engineer/Consultant to ensure quality, reliability, and compliance of AI/ML and GenAI solutions within Pfizer's pharmaceutical environment.

The role emphasizes model evaluation, test strategy, and automation, collaborating with data scientists, ML engineers, and product stakeholders. The ideal candidate has 5+ years in AI/ML QA, strong Python scripting, and experience with LangChain/LangGraph.