Senior AI Evaluations & Annotations PM

Royper Royco LLC; New York, San Francisco (NY, CA); Remote; USD 200,000 - 250,000; Full time

Benefits offered by this job

WFH monthly stipend

Office options in NYC/SF

Job summary

Arize AI is seeking a highly technical product-minded PM to own Evals & Annotations, managing the full evaluation lifecycle from creation to human review and iteration. You will collaborate with Design and Engineering to deliver a best-in-class developer experience that supports evolving AI workflows.

You’ll shape the future of evaluations in an era of autonomous agents, working closely with our cofounders and customers to validate problems and opportunities.

Qualifications

• Significant product management experience owning developer-facing products.
• Deep understanding of LLM evaluations and AI workflows.
• Experience with developer tools and AI infrastructure.
• Hands-on programming experience and ability to prototype.
• Strong collaboration with design and engineering teams.

Responsibilities

• Own Evals & Annotations product area end-to-end.
• Set strategy and roadmap balancing customer needs and UX improvements.
• Prototype solutions and partner with Design to ship polished experiences.
• Shape the future of evaluations as AI development evolves toward autonomous agents.
• Collaborate with Engineering on backend systems, orchestration, and data infrastructure.

Skills

Product management

LLM evaluations

Developer tools

Backend systems

Prototyping

Engineering collaboration

Tools

AI infrastructure

ML platforms

Distributed systems

Job description

Arize AI is seeking a highly technical product-minded PM to own Evals & Annotations, managing the full evaluation lifecycle from creation to human review and iteration. You will collaborate with Design and Engineering to deliver a best-in-class developer experience that supports evolving AI workflows.

You’ll shape the future of evaluations in an era of autonomous agents, working closely with our cofounders and customers to validate problems and opportunities.