Senior AI Product Manager, Evals & Annotations

arizeai; United States; On-site; USD 130,000 - 170,000; Full time

Job summary

Arize AI is seeking a highly technical product manager to own Evals & Annotations, guiding the end-to-end evaluation lifecycle from creation to human review and analysis. You will define the product strategy, prototype solutions with design, and collaborate with engineering to deliver a polished developer experience while shaping the future of evaluations in autonomous and agent-based AI.

This role blends hands-on product thinking with practical execution, working with cofounders and

Qualifications

• Proven track record owning technically complex developer-facing products.
• Strong understanding of LLM evaluations and AI development workflows.
• Experience with developer tools, AI infrastructure, ML platforms, and backend systems.

Responsibilities

• Own Arize's Evals & Annotations product area end-to-end: creation, execution, human review, and annotation workflows.
• Set the product strategy and roadmap, balancing customer needs, platform investments, UX improvements, and AI opportunities.
• Prototype solutions independently and partner with Design to deliver intuitive, polished experiences.
• Shape the future of evaluations as AI evolves toward autonomous and managed agents, including Dynatrace ecosystem opportunities.
• Collaborate with Engineering on backend systems, orchestration, and data infrastructure; discuss tradeoffs and approaches.
• Spend significant time with customers to understand evaluation methods, problems, and validate direction.
• Prioritize across competing demands and drive products from problem definition through launch and iteration.

Skills

Product management

Developer-facing products

LLM evaluations

AI infrastructure

ML platforms

Cross-functional collaboration

Job description

About Arize

AI is rapidly transforming the world. As generative AI reshapes industries, teams need powerful ways to monitor, troubleshoot, and optimize their AI systems. That's where we come in. Arize AI is the leading AI & Agent Engineering observability and evaluation platform, empowering AI engineers to ship high-performing, reliable agents and applications. From first prototype to production scale, Arize unifies build, test, and run in a single workspace, so teams can ship faster with confidence.

Arize serves a rapidly growing customer base of 150+ leading enterprises and Fortune 500 companies. Customers like Booking.com , Uber, Siemens, and PepsiCo leverage Arize to deliver AI that works.

The Opportunity

Evaluations are central to how AI developers understand, measure, and improve their applications. As AI systems become increasingly complex and agentic, developers need cohesive workflows to define what good looks like, evaluate behavior, incorporate human feedback, and continuously improve their applications.

We're looking for a highly technical, product-minded PM to own Evals & Annotations at Arize. You'll own the end-to-end evaluation lifecycle-from eval creation and execution to human review, analysis, and iteration. This is an opportunity to bring together powerful capabilities into a cohesive, best-in-class developer experience while helping define the future of evaluations in a world of increasingly autonomous and managed agents, including through our integration with Dynatrace.

This role is ideal for someone with strong product and UX opinions who doesn't just define requirements but actively shapes the experience. You'll independently explore workflows, build prototypes, and collaborate closely with Design and Engineering to turn ideas into polished, technically sound products. You should be as comfortable working through an interaction design as you are discussing technical tradeoffs with engineers.

While Arize has been acquired, we continue to operate with the urgency, resourcefulness, and hands-on mentality of an early-stage startup. Product, Engineering, and Design collaborate closely, and our founders remain actively involved in product thinking and design. You'll have real ownership while working directly with our cofounder/CPO in an environment that values initiative, constructive debate, and a willingness to jump in wherever needed.

What You’ll Do

• Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle.
• Set the product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development.
• Drive a best-in-class evaluation experience. Develop strong opinions about how workflows should work, independently prototype solutions, and partner with Design to turn those ideas into intuitive, polished experiences.
• Shape the future of evaluations as AI development evolves toward autonomous and managed agents, including opportunities within the broader Dynatrace ecosystem.
• Work closely with Engineering on technically complex problems spanning evaluation execution, backend systems, orchestration, and data infrastructure. Contribute to technical discussions, implementation approaches, and tradeoffs.
• Spend significant time with customers-typically multiple conversations each week-to understand how they evaluate and improve AI systems, uncover underlying problems, and validate product direction.
• Collaborate directly with our cofounders and cross-functional teams on product direction, design exploration, and prioritization, bringing independent judgment and turning collaborative thinking into execution.
• Make clear prioritization decisions across competing demands and drive products from problem definition through launch, adoption, and iteration.

What We’re Looking For

• Significant product management experience independently owning technically complex developer-facing products. We're flexible on exact years of experience for candidates with exceptional technical depth, product judgment, and ownership.
• Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and how developers iterate on AI applications and agents.
• Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products, with a solid understanding of backend systems and software architecture.

Strong technical depth and hands-on engineering ability. We don't expect you to ship large features as an IC engineer, but we do expect you to:

• Debug issues independently and navigate code when needed.
• Put up small fixes or PRs.Build prototypes and technical proofs of concept.
• Propose technical implementations and reason through tradeoffs.
• Hold credible technical discussions with engineers.

Excellent product and UX instincts. You can i