AI Platform PM: Evaluations & Agentic Testing

Relativity; Illinois; Hybrid; USD 115,000 - 173,000; Full time

Job summary

Relativity is seeking a Product Manager for the Evaluations Platform to drive the vision and roadmap across offline/online evaluation, production monitoring, and agentic testing. You will collaborate with applied scientists, engineers, and PMs to help teams measure AI quality and accelerate shipping of capabilities.

The role requires 4+ years in product management on technical or ML products, strong communication, and the ability to translate technical insights into actionable roadmaps.

Qualifications

• 4+ years of product management experience at a tech company, including 2+ years on technical/platform/data/ML products.
• Strong understanding of the software development lifecycle and agile practices.
• Experience conducting user discovery with technical audiences and translating findings into roadmap decisions.
• Excellent written and verbal communication, with ability to explain complex technical concepts to technical and non-technical audiences.

Responsibilities

• Own the product vision, strategy, and roadmap for the Evaluations Platform across its pillars.
• Collaborate with applied scientists, application engineering teams, and PMs to measure and improve AI quality without routing every question through an applied scientist.
• Lead the orchestration of a world-class agentic testing system including evaluation, tool-call correctness, rubrics, and multi-step flows.
• Define SME authoring and approval workflow so experts can encode, score, and reuse judgments across products.

Skills

LLM evaluation

Platform PM

Technical storytelling

Analytical mindset

Agile practices

Stakeholder management

Team leadership

Job description

Relativity is seeking a Product Manager for the Evaluations Platform to drive the vision and roadmap across offline/online evaluation, production monitoring, and agentic testing. You will collaborate with applied scientists, engineers, and PMs to help teams measure AI quality and accelerate shipping of capabilities.

The role requires 4+ years in product management on technical or ML products, strong communication, and the ability to translate technical insights into actionable roadmaps.