Artificial Intelligence Engineer

SIRO Clinpharm Pvt. Ltd.; United States; À distance; USD 120 000 - 180 000; Plein temps

Résumé du poste

SIRO Clinpharm Pvt. Ltd. is seeking an AI QA Engineer/Consultant to ensure quality, reliability, and compliance of AI/ML and GenAI solutions within Pfizer's pharmaceutical environment.

The role emphasizes model evaluation, test strategy, and automation, collaborating with data scientists, ML engineers, and product stakeholders. The ideal candidate has 5+ years in AI/ML QA, strong Python scripting, and experience with LangChain/LangGraph.

Qualifications

• Must have 5+ years in AI/ML, GenAI, model QA/validation, software QA, data science, or quality engineering.
• Experience preferred in regulated or quality-critical environments.

Responsabilités

• Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle.
• Evaluate ML models using metrics like accuracy, precision, recall, F1, ROC/AUC.
• Test models across datasets, edge cases, failure scenarios, and operating conditions.
• Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination, groundedness.
• Design testing for AI agents and multi-agent workflows, including orchestration and tool calling.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
• Use Python, APIs, scripting, and automation for AI testing and evaluation.
• Work with LangChain, LangGraph, Langfuse for agent development and observability.
• Identify bias, drift, hallucination, instability; perform root-cause analysis.
• Define test cases, user stories, acceptance criteria, and production-readiness criteria.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, product owners.
• Document results and risks in audit-ready format; support Responsible AI initiatives.

Connaissances

ML Model Evaluation

GenAI / LLM Evaluation

Agentic AI

Test Strategy

Python Automation

LangChain / LangGraph

Prompt Engineering

AI Risk & Bias

Model Regression Testing

Stakeholder Management

Description du poste

AI QA Engineer/Consultant

Remote

Experience:

5+ years in AI/ML, GenAI, Model QA/Validation, Software QA, Data Science, or Quality Engineering, preferably with experience in regulated or quality-critical environments.

Role Overview: Responsible for ensuring the quality, reliability, performance, and compliance of AI/ML, GenAI, LLM, and Agentic AI solutions within Pfizers pharmaceutical and manufacturing environment.

Key Responsibilities

• Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle.
• Evaluate ML models using Accuracy, Precision, Recall, F1, ROC/AUC, Confusion Matrix, and other relevant metrics.
• Test models across datasets, edge cases, failure scenarios, and different operating conditions.
• Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination, groundedness, and response quality.
• Design and execute testing for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
• Use Python, APIs, scripting, and automation for AI testing and evaluation.
• Work with LangChain, LangGraph, Langfuse and similar tools for agent development, tracing, observability, and evaluation.
• Identify bias, drift, hallucination, instability, and model/application failures and perform root-cause analysis.
• Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and Business Stakeholders.
• Document test results, model limitations, risks, defects, and evaluation outcomes in an audit-ready format.
• Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices.

Must-Have Skills

• Machine Learning & Model Evaluation
• GenAI / LLM Evaluation
• Agentic AI & Multi-Agent Systems
• Precision, Recall, F1, ROC/AUC, Confusion Matrix
• Test Strategy & Test Case Design
• Python, APIs & Automation
• LangChain, LangGraph, Langfuse
• Prompt Engineering & LLM Testing
• AI Risk, Bias & Robustness Testing
• Model/AI Application Regression Testing
• Strong Analytical & Stakeholder Management Skills

Good to Have

• Pharmaceutical / Life Sciences experience
• GxP / CSV / Data Integrity knowledge
• Azure / AWS / GCP
• MLOps / LLMOps / ModelOps
• RAG & Vector Databases
• Responsible AI & AI Governance
• CI/CD and automated testing
• AI Red-Teaming / Adversarial Testing