Lead, AI Evals & Annotations Platform

arizeai; United States; On-site; USD 130,000 - 170,000; Full time

Job summary

Arize AI is seeking a highly technical product manager to own Evals & Annotations, guiding the end-to-end evaluation lifecycle from creation to human review and analysis. You will define the product strategy, prototype solutions with design, and collaborate with engineering to deliver a polished developer experience while shaping the future of evaluations in autonomous and agent-based AI.

This role blends hands-on product thinking with practical execution, working with cofounders and

Qualifications

• Proven track record owning technically complex developer-facing products.
• Strong understanding of LLM evaluations and AI development workflows.
• Experience with developer tools, AI infrastructure, ML platforms, and backend systems.

Responsibilities

• Own Arize's Evals & Annotations product area end-to-end: creation, execution, human review, and annotation workflows.
• Set the product strategy and roadmap, balancing customer needs, platform investments, UX improvements, and AI opportunities.
• Prototype solutions independently and partner with Design to deliver intuitive, polished experiences.
• Shape the future of evaluations as AI evolves toward autonomous and managed agents, including Dynatrace ecosystem opportunities.
• Collaborate with Engineering on backend systems, orchestration, and data infrastructure; discuss tradeoffs and approaches.
• Spend significant time with customers to understand evaluation methods, problems, and validate direction.
• Prioritize across competing demands and drive products from problem definition through launch and iteration.

Skills

Product management

Developer-facing products

LLM evaluations

AI infrastructure

ML platforms

Cross-functional collaboration

Job description

Arize AI is seeking a highly technical product manager to own Evals & Annotations, guiding the end-to-end evaluation lifecycle from creation to human review and analysis. You will define the product strategy, prototype solutions with design, and collaborate with engineering to deliver a polished developer experience while shaping the future of evaluations in autonomous and agent-based AI.

This role blends hands-on product thinking with practical execution, working with cofounders and