Product Manager Evaluation & Data (EC AI)

Rakuten Asia Pte Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Rakuten Asia Pte. Ltd. in Singapore is seeking a Product Manager – Evaluation and Data to define how AI product quality is measured and improved. You will own the evaluation strategy, coordinating with product, engineering, data science, analytics, and business teams.

The role focuses on building scalable evaluation platforms, automating graders, developing data pipelines, and governance to guide release decisions and KPIs.

Qualifications

• 5+ years in AI/ML product management or data platforms.
• Strong analytical skills with large datasets.
• Ability to define platform requirements and data pipelines.
• Experience with A/B testing and production monitoring.
• Excellent written and verbal communication.
• Proven cross-functional leadership.

Responsibilities

• Define evaluation strategy and measurable success criteria.
• Design and calibrate graders and review transcripts.
• Define platform roadmap with engineering and data teams.
• Prioritize datasets, model/prompt evaluation tooling, and monitoring capabilities.
• Connect evaluations with production monitoring and business KPIs.

Skills

AI/ML PM

Data analysis

Platform requirements

A/B testing

Cross-functional leadership

Communication

Job description

Rakuten Group, Inc. is a global leader in internet services and has a diverse ecosystem spanning across e-commerce, fintech, communications and more serving approximately 1.8 billion members worldwide. Founded in Tokyo in 1997, the Group operates in over 30 countries and regions with more than 30,000 employees.

Rakuten Group, Inc. is a global leader in internet services and has a diverse ecosystem spanning across e-commerce, fintech, communications and more serving approximately 1.8 billion members worldwide. Founded in Tokyo in 1997, the Group operates in over 30 countries and regions with more than 30,000 employees.

Based in Singapore's Central Business District, Rakuten Asia Pte. Ltd. serves as the regional headquarters for Asia, driving value through areas such as advertising product development, product strategy, and data management to support Rakuten Group's global ecosystem.

Learn more at: https://global.rakuten.com/corp/

As the Product Manager – Evaluation and Data, your mission is to make the quality of Rakuten's AI products measurable, repeatable, and actionable. You will own the evaluation strategy, process, and platform that enable teams to define success, prevent regressions, make confident release decisions, and continuously improve AI experiences through reliable data and insights.

• High Impact Role: You'll establish how AI product quality is measured, directly influencing release decisions, user experience, and business growth.
• Evaluation-Driven Culture: You'll turn product requirements and real-world failures into reliable evaluations with talented engineers, data scientists, and domain experts.
• Ownership & Autonomy: You'll own the evaluation strategy and platform roadmap, making critical decisions on processes, capabilities, quality standards, and priorities across Rakuten's AI products.
• Collaborative Environment: You'll work closely with product, engineering, data science, analytics, and business teams to define success and achieve shared goals.

Position Details

• Evaluation Strategy & Criteria: Define success for Rakuten's AI products and translate requirements, user needs, and risks into measurable criteria.
• Evaluation Quality: Select grading methods, calibrate automated graders with human judgment, and review transcripts and failures.
• Platform Strategy: Define the platform vision, roadmap, and requirements with engineering and data teams.
• Platform Capabilities: Prioritize dataset management, model/prompt comparison, automated and human grading, multi-turn evaluation, trace review, dashboards, and release integration.
• Continuous Improvement: Maintain balanced suites, convert production issues into regression tests, and improve evaluation coverage and accuracy.
• Adoption & Decisions: Establish governance, enable team and expert contributions, and connect evaluations with production monitoring, A/B testing, feedback, and business KPIs to guide priorities and releases.

Mandatory Qualifications

• Experience: 5+ years of product management experience in AI/ML, evaluation, data products, or platforms. Experience with consumer AI products is preferred.
• Data Proficiency: Excellent analytical skills and experience working with large datasets.
• Platform & Technical Acumen: Ability to define platform requirements and communicate model/prompt versioning, tracing, data pipelines, and release integration.
• Experimentation & Quality: Experience with A/B testing, production monitoring, user feedback, and human evaluation.
• Communication Skills: Excellent written and verbal skills, with the ability to present complex evaluation results clearly and concisely.
• Collaboration Skills: Proven ability to lead cross-functional product, engineering, data science, and domain teams.

Rakuten is an equal opportunities employer and welcomes applications regardless of sex, marital status, ethnic origin, sexual orientation, religious belief, or age.