Senior Product Manager, Evals & Quality – EMEA

PENCIL; Saint-Ouen-l'Aumône; Remote; EUR 90.000 - 120.000; Vollzeit

Benefits dieser Stelle

PTO 25 days + holidays

Health insurance

Wellness stipend

Remote work from home in France

Parental leave

Office access in The Shard (London)

Zusammenfassung

Pencil is seeking a leader to own the Evals & Quality charter, defining success metrics, rubrics, and evaluation processes for our generative AI advertising platform. You will shape how we measure first-pass quality, coverage, and regression, aligning automated judges with human raters and scaling QC across formats and client contexts.

You will also own the golden dataset, its versioning, and the public adevals.ai site, ensuring rigorous evidence for commercial work and client responses.

Qualifikationen

• Experience defining quality metrics for generative AI or ML systems.
• Ability to design rubrics, calibration processes, and evaluation loops.

Aufgaben

• Own the charter for Evals & Quality and define success metrics.
• Define core quality metrics: first-pass-right, coverage, regression detection.
• Own the judging methodology and calibration process between automated and human raters.
• Design and run human QC mechanisms and scaling strategies.
• Own the golden dataset and its versioning across formats and contexts.
• Build eval loops used by other teams and ensure product quality of adevals.ai.

Kenntnisse

Quality measurement

Rubric design

Calibration

Inter‑rater reliability

Data curation

Jobbeschreibung

Pencil is seeking a leader to own the Evals & Quality charter, defining success metrics, rubrics, and evaluation processes for our generative AI advertising platform. You will shape how we measure first-pass quality, coverage, and regression, aligning automated judges with human raters and scaling QC across formats and client contexts.

You will also own the golden dataset, its versioning, and the public adevals.ai site, ensuring rigorous evidence for commercial work and client responses.