Sr. Manager, Test Automation & AI Evaluation

WebMD; Newark (NJ); On-site; USD 98,000 - 135,000; Full time

Benefits offered by this job

Health Insurance

Paid Time Off

401(k) Retirement Plan

Life and Disability Insurance

Employee Assistance Program

Job summary

WebMD Ignite, a division of WebMD and Internet Brands, seeks a Sr. Manager, Test Automation & AI Evaluation to elevate quality across the Ignite Growth Platform. You will champion automation, lead AI-powered QA initiatives, and drive release-readiness across teams.

You will mentor the existing QA team, develop evaluation frameworks for AI features, and partner with engineering, product, and DevOps to embed quality gates throughout the delivery lifecycle.

Qualifications

• 7+ years of quality engineering or test automation experience, including leadership responsibilities.
• Hands-on expertise with modern test automation frameworks (Playwright, Cypress, Selenium).
• Experience building automated test suites and integrating them into CI/CD pipelines.
• Experience designing evaluation methods for AI/ML or LLM-powered systems.
• Strong knowledge of test strategy, defect management, and release governance.

Responsibilities

• Lead an automation-first quality strategy across teams.
• Own modern test automation frameworks and end-to-end automated coverage.
• Build evaluation frameworks for AI-powered features and agentic systems.
• Apply AI to QA, including self-healing automation and AI-assisted test generation.
• Standardize quality engineering practices and drive accountability for release readiness.
• Provide management and mentorship to the Ignite QA function.

Skills

Quality engineering

Test automation

Leadership

AI evaluation

CI/CD integration

Playwright

Cypress

Selenium

Tools

Playwright

Cypress

Selenium

Job description

WebMD Ignite, a division of WebMD and Internet Brands, is the growth partner for healthcare organizations. We guide people to better health at all stages of their journey, from discovery to recovery. Our combination of leading brands in the industry—including WebMD, Medscape, Krames, PulsePoint, Vitals, The Wellness Network, Mercury Healthcare, and Healthwise—offers comprehensive solutions that engage individuals with timely, relevant messaging that optimizes experiences and outcomes, driving loyalty and lifetime value for our clients. Learn more at WebMDIgnite.com.

WebMD Ignite, a division of WebMD and Internet Brands, is the growth partner for healthcare organizations. We guide people to better health at all stages of their journey, from discovery to recovery. Our combination of leading brands in the industry — including WebMD, Medscape, Krames, PulsePoint, Vitals, The Wellness Network, Mercury Healthcare, and Healthwise — offers comprehensive solutions that engage individuals with timely, relevant messaging that optimizes experiences and outcomes, driving loyalty and lifetime value for our clients. Learn more at WebMDIgnite.com.

About the Role

The Sr. Manager, Test Automation & AI Evaluation is an automation-first quality leader responsible for quality engineering across the Ignite Growth Platform. As engineering velocity has accelerated with AI-assisted development, quality assurance has become the rate-limiting constraint; this role exists to remove that constraint. The Sr. Manager makes automation the default, builds the evaluation discipline required to test AI-powered and agentic features, and applies AI to the practice of QA itself. This role also provides direct leadership of the existing Educate quality assurance team and serves as the head of quality assurance across Ignite, owning quality standards, test strategy, and release-readiness governance organization-wide.

Key Responsibilities

• Lead an automation-first quality strategy, eliminating manual testing waste and making automated testing the default across teams.
• Own modern test automation frameworks and end-to-end automated coverage, with deep, current expertise in Playwright and comparable frameworks.
• Build and own evaluation frameworks for LLM-powered features and agentic systems, covering correctness, hallucination, prompt drift, and behavioral regression.
• Establish and maintain evaluation methodology such as curated golden sets, LLM-as-judge approaches calibrated against subject-matter experts, and continuous learning loops that feed production failures back into the benchmark.
• Apply AI to QA itself, including self-healing automation, AI-assisted test generation, and intelligent defect triage.
• Own standardization of quality engineering practices across the organization, driving accountability for test coverage, defect management, release readiness, and overall delivery quality.
• Provide direct management, mentorship, and structure to the existing Ignite quality assurance function.
• Support the development of a scalable Quality Management System to ensure consistency, traceability, and process rigor across teams.
• Partner with engineering, product, and DevOps to define acceptance criteria and embed quality gates into the delivery lifecycle.

Qualifications

• 7+ years of experience in quality engineering or test automation, including team leadership or management.
• Deep, current, hands-on expertise in modern test automation frameworks such as Playwright, Cypress, or Selenium.
• Demonstrated experience building automated test suites and integrating them into CI/CD pipelines.
• Experience designing evaluation or testing approaches for AI/ML or LLM-powered systems, or a strong data-and-metrics orientation with the ability to build one.
• Strong understanding of test strategy, defect management, and release-readiness governance across multiple teams.
• Proven ability to lead, mentor, and grow a quality engineering team.
• Excellent communication skills and the ability to translate quality risk into business terms.

Preferred Skills

• Experience testing generative AI or agentic products, including evaluation harnesses, prompt testing, and regression testing for AI flows.
• Experience in a healthcare, health IT, or other regulated delivery environment where quality standards are elevated.
• Familiarity with quality management systems and traceability requirements.
• Experience with AI-assisted testing tooling and self-healing automation platforms.

Compensation

$98,000 - $135,000

Bonus Eligible: This position is also eligible for a discretionary company bonus, based upon business results.

Benefits

Employees in this position are eligible to participate in the company sponsored benefit programs, including the following within the first 12 months of employment:

• Health Insurance (medical, dental, and vision coverage)
• Paid Time Off (including vacation, sick leave, and flexible holiday days)
• 401(k) Retirement Plan with employer matching
• Life and Disability Insurance
• Employee Assistance Program (EAP)
• Commuter and/or Transit Benefits (if applicable)

Eligibility for specific benefits may vary based on job classification, schedule (e.g., full-time vs. part-time), work location and length of employment.