Senior AI Model Evaluation Consultant - Regulatory Compliance

Ernst & Young LLP ( EY India ); Bengaluru; In loco; INR 1.800.000 - 3.000.000; Tempo pieno

Descrizione del lavoro

EY is seeking an experienced AI/ML model evaluation consultant to validate GenAI systems in regulated life sciences contexts. You will lead evaluation across the model lifecycle, embedding Responsible AI and quality‑by‑design practices, while partnering with data scientists, engineers, and QA teams to ensure regulatory compliance and robust, explainable AI.

The role emphasizes risk identification, audit‑ready deliverables, and strong stakeholder communication, with a focus on GxP alignment and

Competenze

• Bachelor’s or master’s degree in Life Sciences, Engineering, or related field.
• 5+ years of experience in AI/ML, GenAI, software testing, validation, or quality engineering roles.
• Experience creating user stories, test scripts, validation plans, or AI solution prototypes.
• Exposure to cloud ecosystems such as Azure, AWS, or GCP.
• Consulting or client‑facing experience preferred.

Mansioni

• Support AI/ML strategies with analysis, validation insights, and readiness assessments.
• Conduct AI model evaluation across development, training, testing, and deployment.
• Assess model performance with metrics like precision, recall, accuracy, F1-score, and confusion matrices.
• Evaluate model behavior for bias, drift, and instability across datasets and edge cases.
• Demonstrate expertise in supervised and unsupervised learning models and evaluation methods.
• Hands‑on evaluation of GenAI/LLM systems, including prompt behavior and output quality.
• Design and execute testing for AI agents and multi‑agent workflows, including orchestration logic and tool use.
• Collaborate with data scientists and engineers to identify improvements and risk mitigations.
• Document evaluation results, limitations, risks, and acceptance criteria clearly for audits.

Conoscenze

AI/ML concepts

GenAI workflows

Model evaluation techniques

LangChain/LangGraph/Langfuse

Regulatory compliance

Formazione

Bachelor's or master's in Life Sciences or Engineering

Strumenti

Azure

AWS

GCP

Version control

Descrizione del lavoro

RC-REGULATORY COMPLIANCE-AI-Senior At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world.

EY- GDS Consulting – Enterprise Risk (ER) – Regulatory Compliance –Senior – GxP AI/Model Evaluation Consultant

As part of our EY-ER- Regulatory Compliance team, you will help the clients by understanding their business needs and delivering solutions in accordance with the EY guidelines & methodologies. As a Regulatory Compliance senior actively establish, maintain and strengthen internal and external relationships. In line with our commitment to quality, you’ll consistently drive projects to completion with high quality deliverables, achieve operational efficiency, proactively identify and raise risks with the client as well as EY senior management team and lead internal initiatives.

The opportunity

We are seeking a highly experienced and detail‑oriented AI Model Evaluation Consultant with strong expertise in validating AI/ML models, Generative AI (GenAI) systems, and multi‑agent workflows within GxP‑regulated and quality‑critical environments. The ideal candidate will ensure the reliability, fairness, robustness, and regulatory compliance of AI solutions while supporting cross‑functional teams in designing, testing, and maintaining production‑ready, inspection‑ready AI systems. This role is responsible for leading AI model evaluation and validation activities across the full model lifecycle, including model development, training, testing, deployment, and post‑deployment monitoring, in alignment with GxP principles, risk‑based validation, and data integrity expectations. The consultant will play a key role in embedding Responsible AI, quality‑by‑design, and governance‑by‑design practices, enabling the safe, trustworthy, and explainable use of AI and agent‑based solutions in regulated life sciences and pharmaceutical contexts.

Your key responsibilities

• Support development of AI/ML and Agentic AI strategies by providing analysis, validation insights, and readiness assessments.
• Perform AI/ML model evaluation and validation across development, training, testing, and deployment phases.
• Evaluate model performance using precision, recall, accuracy, F1‑score, and confusion matrices.
• Assess model behavior across datasets, edge cases, and failure scenarios to identify bias, drift, and instability.
• Demonstrate strong understanding of supervised and unsupervised learning models, including classification, regression, clustering, and anomaly detection.
• Conduct hands‑on evaluation of GenAI and LLM‑based systems, including prompt behavior, response quality, and consistency.
• Design and execute testing strategies for AI agents and multi‑agent workflows, covering orchestration logic, tool invocation, and output validation.
• Use LangChain, LangGraph, and Langfuse for agent workflows, observability, traceability, and evaluation logging.
• Collaborate with data scientists and engineers to identify model improvement opportunities and risk mitigations.
• Document evaluation results, limitations, risks, and acceptance criteria in a clear, audit‑ready manner.

Qualifications

• Bachelor’s or master’s degree in Life Sciences, Engineering, or related field.
• 5+ years of experience in AI/ML, GenAI, software testing, validation, or quality engineering roles.
• Experience creating user stories, test scripts, validation plans, or AI solution prototypes.
• Exposure to cloud ecosystems such as Azure, AWS, or GCP.
• Consulting or client‑facing experience preferred.
• Excellent communication, documentation, and stakeholder management skills.

Must-Have Skills And Attributes

• Strong understanding of ML models, LLMs, GenAI workflows, agentic systems, and evaluation techniques.
• Strong foundation in machine learning concepts and model evaluation techniques.
• Hands‑on experience with: Precision, Recall, Confusion Matrix, ROC/AUC, Supervised and Unsupervised ML models.
• Practical experience with LangChain, LangGraph, and Langfuse.
• Experience evaluating GenAI / LLM‑based applications and agents.
• Strong documentation, analytical, and stakeholder communication skills.
• Experience designing test cases, user stories, and acceptance criteria for AI/ML‑based systems.
• Ability to validate AI outputs, evaluate accuracy, detect bias, and assess model reliability under varied conditions.
• Hands‑on skills in building prototypes, test harnesses, and demo environments.
• Strong stakeholder management, analytical thinking, and structured communication skills.
• Familiarity with version control, prompt engineering fundamentals, and AI‑assisted testing tools is preferred.
• Strong knowledge of GxP regulations (FDA 21 CFR Part 11, EU Annex 11) GAMP 5 methodology and system development life cycle (SDLC) CSV documentation: URS, FRS, IQ/OQ/PQ, Traceability Matrix, Validation Plans.
• Understanding of data integrity principles and electronic records/e‑signatures compliance.
• Ability to collaborate with cross‑functional teams (QA, IT, Manufacturing, Labs).
• Experience in client‑facing roles, managing expectations and delivering solutions.
• Strong communication and presentation skills.
• Experience in IT project management and validation planning.
• Ability to troubleshoot application issues and recommend solutions.
• Familiarity with change control, deviation handling, and CAPA processes.
• Proven ability to mentor junior staff and lead validation teams.
• Experience in training and cross‑skilling resources.
• Strong teamwork and collaboration mindset.

Good-to-Have Skills And Attributes

• Exposure to business development activities and client engagement strategies.
• Ability to create innovative insights and contribute to thought leadership.
• Understanding of market trends and competitor landscape.
• Experience in process optimization and operational efficiency initiatives.
• Ability to align technology with business transformation goals.
• High attention to detail and analytical thinking.
• Adaptability to changing regulatory and business environments.
• Strong problem-solving and decision-making abilities.

What we look for

A Team of people with commercial acumen, technical experience and enthusiasm to learn new things in this fast-moving environment with consulting skills. An opportunity to be a part of market‑leading, multi‑disciplinary team of 1400 + professionals, in the only integrated global transaction business worldwide. Opportunities to work with EY Consulting practices globally with leading businesses across a range of industries

Working at EY offers

• Support, coaching and feedback from some of the most engaging colleagues around
• Opportunities to develop new skills and progress your career
• The freedom and flexibility to handle your role in a way that’s right for you

EY | Building a better working world

EY is building a better working world by creating new value for clients, people, society and the planet, while building trust in capital markets. Enabled by data, AI and advanced technology, EY teams help clients shape the future with confidence and develop answers for the most pressing issues of today and tomorrow. EY teams work across a full spectrum of services in assurance, consulting, tax, strategy and transactions. Fueled by sector insights, a globally connected, multi‑disciplinary network and diverse ecosystem partners, EY teams can provide services in more than 150 countries and territories.