AI Quality Platform Product Manager

Ridgeline; New York (NY); On-site; USD 136,000 - 187,000; Full time

Benefits offered by this job

Unlimited vacation

Education reimbursement

Wellness reimbursement

0-cost insurance

Job summary

Collaborate to build evaluation frameworks, establish quality benchmarks, and translate model advances into platform capabilities that generate real customer value. Strong analytical and communication skills are essential in a fast-moving, innovation-driven environment.

Qualifications

• 5+ years of product management experience building highly technical platform, infrastructure, or developer-focused products.
• Shipping production AI products powered by large language models.
• Experience defining products that incorporate LLM evaluation frameworks, AI quality measurement, and continuous model evaluation.
• Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics.
• Experience with LLM observability platforms such as Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo, or similar AI quality and evaluation solutions.

Responsibilities

• Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform.
• Build scalable LLM evaluation frameworks and automated evaluation harnesses for continuous model validation.
• Establish standardized offline and online evaluation methodologies, golden datasets, and quality benchmarks across AI products.
• Define the roadmap for AI observability to monitor quality, latency, cost, reliability, and user outcomes in production.
• Partner with engineering and AI researchers to build regression testing and experimentation for generative AI systems.
• Drive product decisions using AI quality metrics, customer feedback, and quantitative evaluation data.
• Collaborate with engineering teams to deliver platform capabilities for trusted, measurable AI experiences.
• Translate advances in foundation models and AI tooling into platform capabilities that create customer value.
• Prioritize investments using customer insights and measurable business outcomes.
• Communicate vision, roadmap, and priorities clearly across technical and executive stakeholders.
• Foster a collaborative environment rooted in ownership and learning.

Skills

5+ years PM experience

AI platform products

LLM evaluation

Observability platforms

Prompt engineering

Stakeholder collaboration

Strong communication

Tools

Datadog LLM Observability

Langfuse

LangSmith

Arize AI

Braintrust

Galileo

Job description

Collaborate to build evaluation frameworks, establish quality benchmarks, and translate model advances into platform capabilities that generate real customer value. Strong analytical and communication skills are essential in a fast-moving, innovation-driven environment.