Senior AI Product Manager, Evals & Annotations (Fully Remote)

Arize AI; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Arize AI is seeking a Product Manager to own the Evals & Annotations product area end-to-end in the United States. You will define the roadmap, balance customer needs with platform investments, and partner with Design to craft an exceptional evaluation experience for AI and agent development.

You will shape core product experiences, prototype solutions, and drive UX improvements, while aligning with a fast-paced startup culture. Strong understanding of LLM evaluations is essential.

Qualifications

• Significant product management experience independently owning technically complex developer-facing products.
• Strong understanding of LLM evaluations and AI development workflows, including automated evals and human feedback.
• Excellent product and UX instincts, with the ability to define and prototype core product experiences.

Responsibilities

• Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle.
• Set the product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development.
• Drive a best-in-class evaluation experience by developing strong opinions, prototyping solutions, and partnering with Design.

Skills

Product management

LLM evaluations

Developer-facing

UX instincts

Prototyping

Job description

• Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle.
• Set the product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development.
• Drive a best-in-class evaluation experience by developing strong opinions, prototyping solutions, and partnering with Design.

What We're Looking For:

• Significant product management experience independently owning technically complex developer-facing products.
• Strong understanding of LLM evaluations and AI development workflows, including automated evals and human feedback.
• Excellent product and UX instincts, with the ability to define and prototype core product experiences.

Bonus Points:

• Direct experience with LLM evaluation platforms, annotation systems, or agent orchestration.
• Strong backend or platform engineering background.
• Early-stage or high-growth startup experience.

Arize AI

Arize AI is the leading AI & Agent Engineering observability and evaluation platform, empowering teams to ship high-performing AI systems. Serving 150+ enterprises like Booking.com and Uber, it operates with a startup culture and hands-on mentality.