RC-REGULATORY COMPLIANCE-GxP AI Model Consultant-Sr

EY; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

EY seeks an experienced AI Model Evaluation Consultant to work with GenAI systems, multi-agent workflows, and GxP-regulated environments, ensuring reliability, fairness, robustness, and regulatory compliance across the model lifecycle.

The role leads validation, testing, deployment, monitoring, and governance-by-design practices, collaborating with data scientists and engineers to deliver production-ready, audit-ready AI solutions in life sciences contexts.

Qualifications

• Strong background in AI/ML, GenAI, software testing, validation, or quality engineering.
• Experience creating user stories, test scripts, validation plans, or AI solution prototypes.
• Exposure to cloud ecosystems such as Azure, AWS, or GCP.

Responsibilities

• Develop AI/ML and GenAI strategies with validation insights.
• Perform AI/ML model integration, evaluation and validation across lifecycle.
• Collaborate with data scientists to identify improvements and risks.
• Design and execute testing for AI agents and multi-agent workflows.
• Document evaluation results, risks, and acceptance criteria audit-ready.
• Establish validation plans, traceability matrices, and reports.

Skills

ML models

LLMs

GenAI workflows

Agentic systems

Evaluation techniques

LangChain

LangGraph

Langfuse

AI observability

Bias & fairness

Education

Bachelor's degree in Life Sciences or Engineering

Master's degree in a related field

Tools

LangChain

LangGraph

Langfuse

Job description

Job Summary

We are seeking a highly experienced and detail-oriented AI Model Evaluation Consultant with strong expertise in working alongside AI/ML models, Generative AI (GenAI) systems, and multi-agent workflows within GxP-regulated and quality-critical environments. The ideal candidate will ensure the reliability, fairness, robustness, and regulatory compliance of AI solutions while engaging directly with cross-functional teams in designing, testing, and maintaining production-ready, inspection-ready AI systems.

This role is responsible for leading AI model consultation and development activities across the full model lifecycle, including model integration, training, testing, deployment, and post-deployment monitoring, in alignment with GxP principles, risk-based validation, and data integrity expectations. The consultant will play a key role in embedding Responsible AI, quality-by-design, and governance-by-design practices, enabling the safe, trustworthy, and explainable use of AI and agent-based solutions in regulated life sciences and pharmaceutical contexts.

Your key responsibilities

• Work on development of AI/ML and Agentic AI strategies by providing analysis, validation insights, and readiness assessments.
• Perform AI/ML model integration, evaluation and validation across development, training, testing, and deployment phases.
• Collaborate with data scientists and engineers to identify model improvement opportunities and risk mitigations.
• Design and execute testing strategies for AI agents and multi-agent workflows, covering orchestration logic, tool invocation, and output validation.
• Document evaluation results, limitations, risks, and acceptance criteria in a clear, audit-ready manner.
• Develop or review validation plans, user and functional requirements, risk assessments, requirements traceability matrices, test protocols, test evidence, deviation records, and validation summary reports.
• Establish acceptance criteria for data quality and model performance, including accuracy, sensitivity, specificity, precision, recall, F1 score, calibration, error rates, and other measures appropriate to the use case.
• Execute or independently review functional, integration, regression, negative, boundary, security-role, audit-trail, data-reconciliation, and user-acceptance testing for AI-enabled workflows.
• Challenge model reliability through reproducibility, robustness, bias, subgroup-performance, explainability, stress, and edge-case testing, with clear documentation of limitations and residual risk.
• Define production monitoring for model performance, data and concept drift, anomalous outputs, override patterns, human review, complaints, incidents, and other signals that may trigger investigation or revalidation.
• Evaluate proposed model updates, retraining, prompt changes, knowledge-base changes, infrastructure changes, and vendor releases through formal change control and impact assessment.
• Ability to perform root cause analysis for hallucinations, inaccurate outputs, agent failures, latency issues, and unexpected model behaviour.
• Assess model behavior across datasets, edge cases, and failure scenarios to identify bias, drift, and instability.
• Understand ML fundamentals - strong grasp of classification, regression, clustering, anomaly detection, and model performance metrics (Precision, Recall, F1, AUC).
• Evaluate LLM and GenAI systems - experience measuring accuracy, completeness, relevance, groundedness, faithfulness, consistency, and task completion.
• Analyze drift and model performance - ability to identify and investigate data drift, concept drift, model drift, and performance degradation in production environments.
• Assess RAG-based solutions - understand retrieval pipelines and be able to evaluate retrieval quality, citation accuracy, and response grounding.
• Work with AI observability platforms - experience with AI monitoring, tracing, evaluation logging, dashboards, and observability tools such as Langfuse, LangSmith, Arize, WhyLabs, MLflow, or similar platforms.
• Use LangChain, LangGraph, and Langfuse for agent workflows, observability, traceability, and evaluation logging.
• Implement and use AI observability capabilities for tracing, evaluation logging, dashboards, alerting, failure analysis, and audit-ready evidence across model and agent workflows.

Qualifications

• Bachelor's or master's degree in Life Sciences, Engineering, or related field.
• 5+ years of experience in AI/ML, GenAI, software testing, validation, or quality engineering roles.
• Experience creating user stories, test scripts, validation plans, or AI solution prototypes.
• Exposure to cloud ecosystems such as Azure, AWS, or GCP.
• Consulting or client-facing experience preferred.
• Excellent communication, documentation, and stakeholder management skills.

Must-Have Skills And Attributes

• Strong understanding of ML models, LLMs, GenAI workflows, agentic systems, and evaluation techniques.
• Strong foundation in machine learning concepts and model evaluation techniques.
• Hands-on experience with: Precision, Recall, Confusion Matrix, ROC/AUC, Supervised and Unsupervised ML models
• Practical experience with LangChain, LangGraph, and Langfuse.
• Experience evaluating GenAI / LLM-based applications and agents.
• Strong documentation, analytical, and stakeholder communication skills.
• Experience designing test cases, user stories, and acceptance criteria for AI