Senior AI Evaluations & Annotations Product Lead

Arize; New York, San Francisco (NY, CA); Remote; USD 200,000 - 250,000; Full time

Benefits offered by this job

WFH stipend

Office options in NYC/SF

Job summary

Arize, a leader in AI observability, is seeking a highly technical, product-minded PM to own Evals & Annotations. You’ll shape evaluation workflows end-to-end—from creation to human review and iteration—while collaborating with Design and Engineering to ship polished experiences.

Work with founders and cross-functional peers in a remote‑first environment, with optional offices in NYC and SF. This role emphasizes ownership, prototyping, and pragmatic tradeoffs in a fast-growing startup.

Qualifications

• Significant PM experience owning complex developer-facing products.
• Deep understanding of LLM evaluation workflows and AI development.
• Experience with developer tools, AI infrastructure, ML platforms.
• Hands-on engineering ability and prototype skills.
• Ability to discuss tradeoffs with engineers and run experiments.

Responsibilities

• Own Evals & Annotations product area end-to-end.
• Set product strategy and roadmap balancing needs.
• Prototype solutions and partner with Design for UX.
• Shape future with Dynatrace ecosystem.
• Collaborate with Engineering on backend, orchestration, data.
• Spend time with customers to validate direction.
• Work with cofounders and cross-functional teams.
• Prioritize across demands and drive launches.

Skills

Product management

Dev-facing products

LLM evaluations

UX instincts

Technical depth

Customer communication

Job description

Arize, a leader in AI observability, is seeking a highly technical, product-minded PM to own Evals & Annotations. You’ll shape evaluation workflows end-to-end—from creation to human review and iteration—while collaborating with Design and Engineering to ship polished experiences.

Work with founders and cross-functional peers in a remote‑first environment, with optional offices in NYC and SF. This role emphasizes ownership, prototyping, and pragmatic tradeoffs in a fast-growing startup.