Technical Product Manager – AI Product Operations

KPMG LLP; Philadelphia (Philadelphia County); On-site; USD 140,000 - 180,000; Full time

Job summary

KPMG LLP seeks a Technical Product Manager for AI Product Operations to shape the internal AI monitoring product, drive its roadmap, and lead a cross-functional team. You will design evaluation systems, build real-time dashboards, and partner with audit and risk teams to ensure high-quality, compliant AI workflows.

This role requires deep ML/AI knowledge, strong communication, and a track record of leading high-performing teams within tech or data science environments.

Qualifications

• Minimum eight years in Technical Product Management, Product Operations, or Data Science.
• Strong technical background with a product mindset; deep ML/AI concepts understanding.
• Experience with Generative AI/LLMs, prompt engineering, and evaluation techniques.
• Excellent written and verbal communication for executive stakeholders.

Responsibilities

• Act as a player-coach managing a team of specialists and define a strategic roadmap for AI monitoring.
• Define roadmap for AI monitoring infrastructure and gather stakeholder requirements.
• Lead creation of real-time dashboards to interpret evaluation results and build monitors.
• Drive root-cause analysis and remediation, create scalable runbooks for deployments.
• Partner with product managers and engineering leaders to design evaluators into the product lifecycle and train on results.
• Collaborate with audit methodology, risk, and compliance to ensure AI evaluation flows support audit standards.
• Uphold integrity and professional environment.

Skills

Technical Product Management

Data Science understanding

AI/GenAI evaluation

Leadership of cross-functional teams

Stakeholder communication

Education

Master’s degree preferred

Bachelor’s degree required

Tools

Dashboard tools

Runbooks

Job description

KPMG is currently seeking a Technical Product Manager - AI Product Operations to join our Audit Technology Alliance organization.

Responsibilities:

• Act as a "player-coach" managing a team of specialists, while treating the AI monitoring and evaluation infrastructure as an internal product with a defined strategic roadmap
• Treat the AI monitoring and evaluation infrastructure as an internal product; define its strategic roadmap, gather stakeholder requirements, and drive its continuous improvement
• Lead the creation of real-time dashboards to interpret evaluation results, building complex monitors to capture agent traces, human-in-the-loop decisions, and user feedback
• Drive root-cause analysis and cross-functional remediation for AI system issues (including documenting post-mortems), while creating scalable runbooks (SOPs) to streamline deployments
• Partner with product managers and engineering leaders to design and implement evaluators directly into the product lifecycle, leading training efforts on how to accurately interpret and use the results
• Collaborate closely with audit methodology specialists, risk, and compliance teams to ensure all AI evaluation flows, metrics, and guardrails actively support firm audit quality standards
• Act with integrity, professionalism, and personal responsibility to uphold KPMG’s respectful and courteous work environment

Qualifications:

• Minimum eight years of recent overall professional experience in Technical Product Management, Product Operations, or Data Science, including a proven track record of building and managing high-performing teams
• Master’s degree from an accredited college or university is preferred; minimum of a Bachelor’s degree from an accredited or university is required
• Strong technical background with a distinct product management mindset; you don't need to write production code, but you must deeply understand ML/AI concepts to design sophisticated evaluation products
• Extensive knowledge of Generative AI/LLMs, prompt engineering, and GenAI evaluation techniques (for example, automated scoring, human-in-the-loop review, hallucination detection)
• Demonstrated ability to conceptualize scalable measurement frameworks, define critical KPIs, and design structured workflows out of complex, unstructured AI outputs
• Exceptional written and verbal communication skills, with a proven ability to translate highly technical AI performance metrics into clear business narratives for executive stakeholders
• Applicants must be authorized to work in the U.S. without the need for employment-based visa sponsorship now or in the future; KPMG LLP will not sponsor applicants for U.S. work visa status for this opportunity (no sponsorship is available for H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT or any other employment-based visa)