AI Evaluation Platform Product Manager

Relativity; Chicago (IL); Teletrabalho; USD 115 000 - 173 000; Tempo integral

Resumo da oferta

Relativity is seeking a Product Manager to own the product strategy and roadmap for the Evaluations Platform, helping teams measure, understand, and improve AI performance across Relativity's products. You will work with applied scientists, application PMs, and legal experts to encode evaluation criteria, enable rapid iteration of agentic capabilities, and ship safer, higher‑quality AI at speed.

This platform PM role emphasizes direct leverage with technical customers, build-vs-buy decisions,

Qualificações

• 4+ years of product management at a technology company, with at least 2 years on technical/platform or data/ML products.
• Solid understanding of the software development lifecycle and agile practices.
• Experience conducting user discovery with technical audiences and translating findings into roadmap decisions.
• Excellent written and verbal communication, able to explain complex technical concepts to technical and non-technical audiences.
• Experience with LLM-based products and agentic systems is a plus.
• Experience as a platform, developer-tools, or data/ML PM serving technical customers.

Responsabilidades

• Own the product vision, strategy, and roadmap for the Evaluations Platform across its pillars: offline evaluation pipeline, online evaluation and production monitoring, eval discovery and reuse, SME authoring and approval, and agentic evaluation.
• Serve every team building AI at Relativity. Run continuous discovery with applied scientists, application engineering teams, and product PMs to understand experiences with AI quality today and design the platform to measure and improve performance.
• Lead the orchestration of a world-class agentic testing system: trajectory evaluation, tool-call correctness, intermediate-state rubrics, long-running judge orchestration, and simulation of multi-step flows.
• Work with legal subject-matter experts to define tasks our AI is evaluated against, the outcomes, hard cases, and how judgment criteria are expressed for encoding and reuse.
• Own team-facing quality dashboards to show performance, quality bar standing, and changes since last release with minimal engineering help.
• Lead build-vs-buy evaluations for tracing infrastructure and commercial eval tooling as the vendor landscape shifts.
• Partner with engineering and applied science on judge calibration so LLM-as-judge is trustworthy enough to gate deployments.
• Clearly articulate technical tradeoffs to stakeholders ranging from applied scientists to attorneys to executive leadership.

Conhecimentos

Agile methodology

Cross-functional teamwork

Market research

Market strategy

Product lifecycle

Product management

Stakeholder management

Team leadership

Descrição da oferta de emprego

Relativity is seeking a Product Manager to own the product strategy and roadmap for the Evaluations Platform, helping teams measure, understand, and improve AI performance across Relativity's products. You will work with applied scientists, application PMs, and legal experts to encode evaluation criteria, enable rapid iteration of agentic capabilities, and ship safer, higher‑quality AI at speed.

This platform PM role emphasizes direct leverage with technical customers, build-vs-buy decisions,