EY - GDS Consulting - AIA - Gen AI - Senior

EY; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

EY Global Delivery Services is seeking an AI QA Senior to drive testing for AI and GenAI solutions, with 5–9+ years of experience in AI testing, quality engineering and enterprise AI platforms.

You will design testing strategies, develop automation using Python, validate LLM outputs and ensure safety, bias controls and regulatory compliance across AI services.

Qualifications

• 5–9+ years of total experience in AI/ML, GenAI or applied AI systems.
• Hands‑on experience validating AI-powered applications and enterprise AI platforms.
• Portfolio projects or tutorials alone are not sufficient for qualification.

Responsibilities

• Design, develop, and execute testing strategies for AI, GenAI, Agentic AI and RAG systems.
• Validate LLM outputs for accuracy, relevance, safety, and compliance requirements.
• Develop test frameworks and automation suites for AI services and enterprise AI apps.
• Design evaluation frameworks for RAG systems, vector search, and knowledge AI apps.
• Perform functional, integration, regression, performance, reliability, and scalability testing of AI apps.
• Develop automation using Python to validate AI workflows, APIs, models, and orchestration pipelines.
• Validate prompt engineering, agent workflows, tool usage patterns, and reasoning in Agentic AI.
• Execute AI model testing across conversational and multimodal AI and intelligent automation.

Skills

AI QA

AI testing

Python

API testing

CI/CD testing

Security testing

Evaluation frameworks

Responsible AI

Performance testing

Education

BTech/Masters/PhD

Tools

Python

FastAPI

Azure AI

Ragas

Retrieval pipelines

Job description

Job Description

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

AI QA – Senior

Educational qualification: BTech/Masters/PhD

The opportunity

We are seeking an experienced AI QA Engineer with 5–9+ years of professional experience and deep expertise in AI testing, quality engineering, validation frameworks, and enterprise AI solution assurance. The ideal candidate should possess strong hands‑on experience validating AI systems, Agentic AI workflows, RAG applications, conversational AI solutions, and enterprise‑scale AI platforms.

Important note: This role requires professional implementation experience in relevant AI technologies. Personal portfolio projects, cloned demos, certifications, or tutorial‑based builds are helpful, but are not sufficient on their own.

Your Key Responsibilities

• Design, develop, and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems.
• Validate Large Language Model outputs for accuracy, consistency, relevance, safety, and compliance requirements.
• Develop test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications.
• Design and execute evaluation frameworks for RAG systems, retrieval pipelines, vector search solutions, and knowledge‑based AI applications.
• Perform functional, integration, regression, performance, reliability, and scalability testing of AI‑powered applications.
• Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines.
• Validate prompt engineering approaches, agent workflows, tool usage patterns, and reasoning capabilities in Agentic AI systems.
• Execute AI model testing across conversational AI, multimodal AI, and intelligent automation solutions.
• Perform AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and security validation activities.
• Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance requirements.
• Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms.
• Utilize RAG evaluation frameworks such as Ragas to measure retrieval quality, answer faithfulness, relevance, and response effectiveness.
• Develop testing dashboards, quality metrics, defect reporting processes, and continuous quality improvement practices.
• Collaborate closely with AI engineers and architects to identify quality risks and improve operational reliability.
• Contribute to quality standards, governance controls, testing accelerators, reusable frameworks, and best practices.
• Stay current with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging quality assurance technologies.

Skills and Attributes

Professional Experience

• 5–9+ years of total experience, including 3+ years of directly relevant hands‑on experience in AI/ML, applied AI, or GenAI systems.
• Proven experience building production, pre‑production, or enterprise pilot AI applications.
• Strong hands‑on depth in at least 3 of the following areas, with meaningful ownership in at least 2:
• LLM application development.
• Retrieval pipelines.
• Agent or tool orchestration.
• Evaluation, guardrails, or quality systems.
• Model adaptation or fine‑tuning.
• Cloud deployment or optimization of AI applications.
• Conversational AI.
• Multimodal AI.
• Backend engineering for AI systems.
• Strong software engineering foundation with ability to build reliable, maintainable AI services.
• Able to clearly explain direct contributions to delivered AI implementations.

Educational Background

• Bachelor’s/master’s degree in computer science, Data Science, Engineering, or related field.

Technical Skills

• Strong understanding of Large Language Models (LLMs), Generative AI, Agentic AI, and Agentic RAG systems.
• Hands‑on experience validating and testing AI‑powered applications and enterprise AI platforms.
• Experience developing AI testing strategies, validation frameworks, and automated test suites.
• Strong expertise in AI Model Testing, functional testing, regression testing, integration testing, and end‑to‑end quality validation.
• Experience performing AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and vulnerability assessments.
• Strong understanding of prompt engineering validation, reasoning evaluation, and response quality assessment techniques.
• Experience implementing evaluation frameworks and quality measurements for Agentic AI workflows and AI agents.
• Hands‑on experience with Ragas and similar evaluation frameworks for assessing retrieval quality, answer correctness, faithfulness, relevance, and grounding.
• Experience validating Agentic RAG systems, retrieval pipelines, and knowledge‑based AI applications.
• Experience testing AI integrations across APIs, enterprise applications, databases, and cloud services.
• Strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking.
• Experience working with Microsoft Azure AI Platform and Azure AI services.
• Hands‑on experience developing automated test frameworks and validation utilities using Python.
• Familiarity with FastAPI‑based services and API testing frameworks.
• Experience designing and executing automated API testing strategies.
• Knowledge of Responsible AI principles, AI safety controls, governance requirements, and regulatory compliance considerations.
• Experience evaluating hallucinations, bias, toxicity, safety violations, and model robustness.
• Strong understanding of software quality engineering, secure development practices, test automation frameworks, and CI/CD testing integration.
• Experience working with performance testing, scalability testing, reliability testing, and production‑readiness validation.
• Familiarity with retrieval augmentation architectures, vector databases, embeddings, and enterprise search solutions.
• Strong analytical, troubleshooting, and root‑cause analysis skills.

Soft Skills

• Excellent problem‑solving skills and ability to connect AI capabilities to business value.
• Strong communication and presentation skills.

Other Responsibilities

• Contribute to end‑to‑end quality assurance activities across AI and GenAI engagements.
• Collaborate with AI engineers, architects, data scientists, and product teams to establish robust testing and validation practices.
• Participate in AI design reviews, risk assessments, model evaluations, and quality governance initiatives.
• Ensure AI solutions meet functional, security, compliance, governance, and business requirements before production deployment.
• Support development of reusable testing assets, automation frameworks, quality accelerators, and evaluation methodologies.
• Contribute to AI governance, Responsible AI, risk management, and compliance initiatives.
• Support presales, innovation initiatives, technical demonstrations, and client workshops where AI testing expertise is required.
• Provide mentorship to junior QA engineers and help establish AI quality engineering best practices.

Why Join Us

• Be at the forefront of AI‑driven innovation across multiple client sectors.
• Work with global clients to drive real business impact.
• Collaborate with a team of AI experts, analytics leaders, and industry specialists in a highly entrepreneurial environment.

What We Offer

EY Global Delivery Services (GDS) is a dynamic and truly global delivery network. We work across six locations – Argentina, China, India, the Philippines, Poland and the UK – and with teams from all EY service lines, geographies and sectors, playing a vital role in the delivery of the EY growth strategy. From accountants to coders to advisory consultants, we offer a wide variety of fulfilling career opportunities that span all business disciplines. In GDS, you will collaborate with EY teams on exciting projects and work with well‑known brands from across the globe. We’ll introduce you to an ever‑expanding ecosystem of people, learning, skills and insights that will stay with you throughout your career.

• Continuous learning: You’ll develop the mindset and skills to navigate whatever comes next.
• Success as defined by you: We’ll provide the tools and flexibility, so you can make a meaningful impact, your way.
• Transformative leadership: We’ll give you the insights, coaching and confidence to be the leader the world needs.
• Diverse and inclusive culture: You’ll be embraced for who you are and empowered to use your voice to help others find theirs.

EY | Building a better working world

EY exists to build a better working world, helping to create long‑term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.