DE-NCLC-SM-Agenetic AI

EY; Bengaluru; On-site; INR 3,500,000 - 5,200,000; Full time

Job summary

EY in Bengaluru is seeking a Senior Manager to own the end-to-end test strategy for the Agentic product, shaping release-quality evidence and acceptance criteria. You will build and maintain automated regression tests with Playwright, Cypress, Selenium and Python tools, and validate complex agent behavior beyond UI/API testing.

The role involves designing diverse test scenarios, safeguarding AI outputs, reviewing logs with SMEs, and coordinating with security and compliance teams to ensure

Responsibilities

• Own the end-to-end test strategy, test plans, acceptance-criteria coverage, and release-quality evidence for the Agentic product.
• Build and maintain automated regression and integration tests using UHG provided tools such as Playwright, Cypress, Selenium, API testing frameworks, and Python testing libraries.
• Validate agent behavior beyond conventional UI/API testing, including tool selection and sequencing, function-call inputs/outputs, orchestration paths, refusal/escalation behavior, and human-in-the-loop controls.
• Design and execute positive, negative, ambiguous, adversarial, and out-of-scope scenarios for every relevant acceptance criterion.
• Maintain evaluation datasets and regression suites for AI-generated outputs, with focus on correctness, consistency, factual grounding, and prevention of regressions when prompts, models, configurations, or tools change.
• Validate agent execution logs and traceability across perception, reasoning, planning, execution, retries/corrections, and resulting outputs so test evidence can be reviewed by SMEs.
• Test security and access-control scenarios, including negative authorization cases and agent/tool boundaries, and coordinate with Security and Compliance SMEs on identified issues.
• Perform defect triage and root-cause analysis with developers, track defects through closure, and provide clear quality and release-readiness reporting through automated scripts/dashboards where practical.
• Support performance and regression testing and contribute evidence required for system testing, UAT, and release gates.

Job description

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

Job Description: Senior Manager

• Own the end-to-end test strategy, test plans, acceptance-criteria coverage, and release-quality evidence for the Agentic product.
• Build and maintain automated regression and integration tests using UHG provided tools such as Playwright, Cypress, Selenium, API testing frameworks, and Python testing libraries.
• Validate agent behavior beyond conventional UI/API testing, including tool selection and sequencing, function-call inputs/outputs, orchestration paths, refusal/escalation behavior, and human-in-the-loop controls.
• Design and execute positive, negative, ambiguous, adversarial, and out-of-scope scenarios for every relevant acceptance criterion.
• Maintain evaluation datasets and regression suites for AI-generated outputs, with focus on correctness, consistency, factual grounding, and prevention of regressions when prompts, models, configurations, or tools change.
• Validate agent execution logs and traceability across perception, reasoning, planning, execution, retries/corrections, and resulting outputs so test evidence can be reviewed by SMEs.
• Test security and access-control scenarios, including negative authorization cases and agent/tool boundaries, and coordinate with Security and Compliance SMEs on identified issues.
• Perform defect triage and root-cause analysis with developers, track defects through closure, and provide clear quality and release-readiness reporting through automated scripts/dashboards where practical.
• Support performance and regression testing and contribute evidence required for system testing, UAT, and release gates.

EY | Building a better working world

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.