Testing - AI Test Staff

Ernst & Young Advisory Services Sdn Bhd; Bengaluru; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

EY Bengaluru is seeking a QE - AI Test Engineer to validate AI-powered solutions across Narrow, Generative, and Agentic AI. You will work with cross-functional teams to ensure quality, performance and trustworthy AI across EY’s Global Delivery Services in India.

The role emphasizes data quality validation, statistical analysis, and applying AI testing techniques using Python-based tools within an Agile environment.

Qualifications

• Proven experience in testing AI models and systems.

Responsibilities

• Validate AI models and systems across Narrow AI, Generative AI, and Agentic AI implementations to ensure quality, reliability, and performance.
• Perform data validation activities, including data collection, data generation, data augmentation, exploratory data analysis, and assessment of data quality, bias, and privacy considerations.
• Evaluate AI model outputs against functional, business, and performance metrics to identify defects, inconsistencies, and improvement opportunities.
• Design and execute test scenarios for AI-based applications using established software testing principles and practices.
• Apply AI-specific testing approaches such as pairwise testing, metamorphic testing, back-to-back testing, bias testing, and drift testing.
• Utilize Python-based frameworks and automation tools to improve testing efficiency and coverage.
• Collaborate with developers, data scientists, product owners, and stakeholders to identify, track, and resolve quality issues throughout the development lifecycle.
• Support validation of Large Language Models (LLMs) and Agentic AI systems using established evaluation methodologies and frameworks.
• Contribute to continuous improvement initiatives by recommending enhancements to testing processes, methodologies, and quality standards.
• Work effectively within Agile teams to support delivery of high-quality AI solutions.

Skills

AI testing

Python

Data validation

Data analysis

Agile

LLMs

Bias testing

Drift testing

Test automation

Quality engineering

Stakeholder management

Communication

Tools

Selenium

Playwright

PyTest

GitHub Copilot

Azure

AWS

GCP

Job description

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

At EY, we’re all in to shape your future with confidence.

We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go.

Join EY and help to build a better working world.

About Global Delivery Services

Global Delivery Services refers to EY's worldwide network of service delivery centers. The GDS team plays an important role in EY’s strategy by ensuring effective support to EY’s growth agenda.

Our journey started in 2002 with approximately 200 people. Today we stand at 80,000+ professionals in ten locations around the world. We operate in Argentina, China, Hungary, India, Philippines, Poland, Sri Lanka, Mexico, Spain and the United Kingdom.

Client service is focused on providing Consulting, Assurance, Tax, Strategy & Transactions, and Knowledge support to our clients around the world. The teams enable account teams worldwide to provide seamless, high-quality, value-added support, helping deliver exceptional client service.

Enablement Services provides cost-effective, high-skilled, and innovative services to support EY’s global and local enablement teams. Markets, BMC, AWS, Finance and Accounting, Risk Management, Procurement, People Shared Services, IT Service Delivery and IT Global Infrastructure services, are among the services offered by Enablement Services.

Our innovation specialists serve the GDS Client Service and Enablement Services teams, along with Service Lines, Core Business Services and Sectors. The team brings the desired environment, technologies and skilled teams together for facilitation, rapid prototyping and innovative thinking. The competencies offered include analytics, digital, user experience, mobile technology, infrastructure, Microsoft technologies and open innovation.

The Opportunity

As a QE - AI Test Engineer, you will play a key role in validating and assuring the quality of AI-powered solutions, including Narrow AI, Generative AI, and Agentic AI systems. You will work closely with cross-functional teams to evaluate AI models, validate data quality, perform statistical analysis, and ensure AI applications meet functional, performance, and business requirements. This role provides an exciting opportunity to work at the forefront of AI innovation while applying quality engineering practices to build reliable, scalable, and trustworthy AI solutions.

Your Key Responsibilities

• Validate AI models and systems across Narrow AI, Generative AI, and Agentic AI implementations to ensure quality, reliability, and performance.
• Perform data validation activities, including data collection, data generation, data augmentation, exploratory data analysis, and assessment of data quality, bias, and privacy considerations.
• Evaluate AI model outputs against functional, business, and performance metrics to identify defects, inconsistencies, and improvement opportunities.
• Design and execute test scenarios for AI-based applications using established software testing principles and practices.
• Apply AI-specific testing approaches such as pairwise testing, metamorphic testing, back-to-back testing, bias testing, and drift testing.
• Utilize Python-based frameworks and automation tools to improve testing efficiency and coverage.
• Collaborate with developers, data scientists, product owners, and stakeholders to identify, track, and resolve quality issues throughout the development lifecycle.
• Support validation of Large Language Models (LLMs) and Agentic AI systems using established evaluation methodologies and frameworks.
• Contribute to continuous improvement initiatives by recommending enhancements to testing processes, methodologies, and quality standards.
• Work effectively within Agile teams to support delivery of high-quality AI solutions.

Skills and attributes for success

• Proven experience in testing AI models and systems
• Strong understanding of software testing principles, methodologies, and quality engineering practices.
• Foundational knowledge of AI/ML concepts, algorithms, and model validation techniques.
• Understanding of Large Language Models (LLMs), Generative AI, and Agentic AI systems.
• Experience with AI testing techniques such as pairwise testing, metamorphic testing, bias testing, drift testing, and back-to-back testing.
• Proficiency in Python programming for test development, validation, and automation activities.
• Experience with test automation tools such as Selenium, Playwright, PyTest, or similar frameworks.
• Knowledge of cloud technologies and platforms including Microsoft Azure, AWS, or Google Cloud Platform.
• Exposure to GitHub Copilot and AI-assisted development tools.
• Strong analytical, troubleshooting, and problem-solving capabilities.
• Excellent communication and stakeholder management skills.
• Ability to collaborate effectively in Agile delivery environments and multidisciplinary teams.

What we offer you

At EY, we’ll develop you with future-focused skills and equip you with world-class experiences. We’ll empower you in a flexible environment, and fuel you and your extraordinary talents in a diverse and inclusive culture of globally connected teams. Learn more.

Are you ready to shape your future with confidence?

To help create an equitable and inclusive experience during the recruitment process, please inform us as soon as possible about any disability-related adjustments or accommodations you may need.

• Our Commitment: As a commitment, we persistently endeavour to embody our values, fulfil our purpose, and champion inclusiveness. Our dedication is to cultivate EY into an environment where diverse perspectives are celebrated, creating a supportive atmosphere for individuals to authentically be themselves and contribute their utmost.
• Professional Development: From entry-level employees to senior leaders, we believe in continuous learning. We offer opportunities to build new skills, take on leadership roles, and connect and grow through mentorship.
• People and Culture: In our dynamic workplace, diversity, equity, and inclusiveness are ingrained in our culture. We're united by a commitment to create an environment where every individual's differences are valued, practices are equitable, fostering a sense of belonging. Our shared values include integrity, respect, teaming, inclusiveness, energy, enthusiasm, courage to lead, and building relationships based on doing the right thing.
• Benefits: Embark on a transformative career journey with us and indulge in a suite of premium benefits, encompassing exclusive health and wellness packages, enticing rewards, and cutting-edge learning opportunities that empower you to continually grow and excel in your professional and personal development.

Successful candidates advance to a competency-based interview. If mutual interest persists, a job offer awaits.

“We are an equal opportunity employer and are committed to Diversity, Equity & Inclusion”.

EY | Building a better working world

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.

EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients.