Senior Manager - Automation and Agentic AI Testing

Ernst & Young LLP ( EY India ); Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

EY India seeks a Senior Manager to own the end-to-end testing strategy for the Agentic AI product. You will lead automated regression and integration tests using Playwright, Cypress, Selenium, and Python testing libraries, validating behavior beyond UI and coordinating with Security for risk mitigation.

As part of EY India's AI testing team, you will design comprehensive test scenarios, manage evaluation datasets, and provide release-readiness evidence across perception, reasoning, and execution

Qualifications

• Own the end-to-end test strategy and release-quality evidence.
• Build and maintain automated regression and integration tests.
• Validate agent behavior beyond UI/API testing.
• Design scenarios and maintain AI evaluation datasets.
• Coordinate with Security and Compliance SMEs on testing issues.

Responsibilities

• Own the end-to-end test strategy, test plans, acceptance-criteria coverage, and release-quality evidence for the Agentic product.
• Build and maintain automated regression and integration tests using Playwright, Cypress, Selenium, API testing frameworks, and Python testing libraries.
• Validate agent behavior beyond conventional UI/API testing, including tool selection and sequencing, function-call inputs/outputs, and human-in-the-loop controls.
• Design positive, negative, ambiguous, adversarial, and out-of-scope scenarios for acceptance criteria.
• Maintain evaluation datasets and regression suites for AI outputs, focusing on correctness and regression prevention when models change.

Skills

Python

Automation

Agentic AI

Test Strategy

Tools

Playwright

Cypress

Selenium

API testing

Python testing libraries

Job description

DE-NCLC-SM-Agenetic AI At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

Job Description:

• Senior Manager Senior Manager stronger Python, automation, agentic AI
• Own the end-to-end test strategy, test plans, acceptance-criteria coverage, and release-quality evidence for the Agentic product.
• Build and maintain automated regression and integration tests using UHG provided tools such as Playwright, Cypress, Selenium, API testing frameworks, and Python testing libraries.
• Validate agent behavior beyond conventional UI/API testing, including tool selection and sequencing, function-call inputs/outputs, orchestration paths, refusal/escalation behavior, and human-in-the-loop controls.
• Design and execute positive, negative, ambiguous, adversarial, and out-of-scope scenarios for every relevant acceptance criterion.
• Maintain evaluation datasets and regression suites for AI-generated outputs, with focus on correctness, consistency, factual grounding, and prevention of regressions when prompts, models, configurations, or tools change.
• Validate agent execution logs and traceability across perception, reasoning, planning, execution, retries/corrections, and resulting outputs so test evidence can be reviewed by SMEs.
• Test security and access-control scenarios, including negative authorization cases and agent/tool boundaries, and coordinate with Security and Compliance SMEs on identified issues.
• Perform defect triage and root-cause analysis with developers, track defects through closure, and provide clear quality and release-readiness reporting through automated scripts/dashboards where practical.
• Support performance and regression testing and contribute evidence required for system testing, UAT, and release gates.

EY | Building a better working world EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate. Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.