AI Quality Platform PM — Observability & Evaluation

Ridgeline, Inc.; New York (NY); On-site; USD 136,000 - 170,000; Full time

Benefits offered by this job

Unlimited vacation

Educational and wellness reimburse- m-

$0 cost employee insurance plans

Job summary

Ridgeline, Inc. is seeking an AI Product Manager for the AI Platform - Quality and Evaluation to define and drive the strategy for our AI quality capabilities.

You will lead initiatives spanning LLM evaluation, model benchmarking, evaluation harnesses, and observability to improve production AI performance across Ridgeline’s products. You will work with engineering, AI researchers, design, and product teams to deliver scalable platform features that quantify AI quality, enable continuous

Qualifications

• 5+ years of product management experience building highly technical platform, infrastructure, or developer-focused products.
• Demonstrated experience shipping production AI products powered by large language models.
• Experience defining products that incorporate LLM evaluation frameworks, AI quality measurement, and continuous model evaluation.
• Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics.
• Experience with LLM observability platforms such as Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo, or similar AI quality and evaluation solutions.

Responsibilities

• Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform.
• Build scalable LLM evaluation frameworks and automated evaluation harnesses that enable continuous model validation.
• Establish standardized offline and online evaluation methodologies, golden datasets, and quality benchmarks across AI products.
• Define the roadmap for AI observability, enabling teams to monitor quality, latency, cost, reliability, and user outcomes in production.
• Partner with engineering and AI researchers to build continuous regression testing and experimentation capabilities for generative AI systems.
• Drive product decisions using AI quality metrics, customer feedback, and quantitative evaluation data.
• Collaborate with engineering teams to deliver platform capabilities that enable trusted, measurable, and continuously improving AI experiences across Ridgeline.
• Translate advances in foundation models, evaluation techniques, and AI tooling into platform capabilities that create meaningful customer value.
• Prioritize product investments using customer insights, product analytics, and measurable business outcomes.
• Communicate product vision, roadmap, and priorities clearly across technical and executive stakeholders.
• Foster a collaborative environment rooted in ownership, continuous learning, experimentation, and transparency.

Job description

Ridgeline, Inc. is seeking an AI Product Manager for the AI Platform - Quality and Evaluation to define and drive the strategy for our AI quality capabilities.

You will lead initiatives spanning LLM evaluation, model benchmarking, evaluation harnesses, and observability to improve production AI performance across Ridgeline’s products. You will work with engineering, AI researchers, design, and product teams to deliver scalable platform features that quantify AI quality, enable continuous