Product Manager, Evals & Improvement

Meta; Menlo Park (CA); Hybrid; USD 180.000 - 240.000; Vollzeit

Zusammenfassung

Meta’s ATA Product team is building Metamate, an internal platform for improving agentic AI across the company. This role focuses on measuring and raising the quality of agentic systems, defining what good means for agents, and owning the eval verdict that gates model upgrades and launches.

The position is early-stage, high-leverage, requiring collaboration with engineering, data science, ML, and partners across ATA and MSL to turn real usage signals into a continuous improvement loop that

Qualifikationen

• 5+ years in industry with at least 2 in product management.
• Experience building evals for AI products end to end: task sets, rubrics, instrumentation.
• Strong quantitative skills and ability to define metrics where none exist.
• Comfort with ambiguous problem spaces and rapid learning.
• Experience with RLHF or human-in-the-loop systems is a plus.

Aufgaben

• Define metrics and the eval verdicts that gate model upgrades and launches.
• Collaborate with data science and ML engineering teams across ATA and MSL.
• Own end-to-end evaluation development: rubric design, instrumentation, analysis.
• Drive measurable impact through continuous improvement loops sprint to sprint.

Kenntnisse

Quantitative analysis

Product management

Experimentation design

AI/ML familiarity

Cross-functional collaboration

Eval metrics

Ausbildung

Bachelor's degree in STEM or related field

Tools

Experimentation platforms

A/B testing

Jobbeschreibung

The Agent Transformation Accelerator (ATA) Product team is building Meta’s future of work: Metamate, an internal full‑stack platform that lets people direct, review, and help agents improve their work across the entire company. We’re building the platform components, memory systems, improvement loops, and shared product experiences that make agentic AI genuinely useful for product development. This role owns how we measure and raise the quality of agentic systems (multi-step trajectories, tool use, and partial credit).

This is an early-stage, high-leverage role that contributes directly to Metamate's topline revenue, growth, and velocity. You'll shape the approach from the ground up in a space where the right metrics don't yet exist, defining what "good" means for agents and owning the eval verdict that gates model upgrades, harness changes, and major launches. Expect to be constantly learning, working shoulder-to-shoulder with engineering, data science, and ML, including our partners across ATA and Meta Superintelligence Labs (MSL), to turn signal from real usage into a continuous improvement loop that makes the product measurably better every week. Strong quantitative skills and experience defining complex metrics Comfort with ambiguous problem spaces where the right metrics do not yet exist 5+ years of relevant industry experience with at least 2 years in Product Management Bachelor's degree (or relevant degree equivalent): STEM subject ideal but not essential (Computer Science, Engineering, Information Systems, Analytics, Mathematics, Physics, Applied Sciences) Experience partnering closely with data science and ML engineering teams Track record of driving product improvements through data and experimentation Direct experience building evals for AI products end to end: task set construction, rubric design, instrumentation, and analysis Experience running eval-driven development cycles, with a clear view of where evals are informative and where they mislead Willingness to get into the weeds of eval data and rubrics, with high standards for eval rigor and an obsession with quality Deep understanding of LLM capabilities and failure modes Experience building annotation, labeling, or crowd-sourcing systems Experience with reinforcement learning from human feedback (RLHF) or similar human-in-the-loop systems Background in evals, trust and safety measurement, or ML quality infrastructure Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements) Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews) Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies.