AI Quality Platform Product Manager

Ridgeline; San Ramon (CA); On-site; USD 136,000 - 170,000; Full time

Benefits offered by this job

Unlimited vacation

Educational reimbursements

Wellness reimbursements

Zero-cost employee insurance plans

Company Stock Plan

Job summary

Ridgeline is seeking a product leader to own the AI Quality Platform, partnering with engineers and researchers to define strategy and execute a roadmap for scalable evaluation, observability, and platform capabilities. You will translate AI performance signals into actionable product decisions and customer value.

The role requires 5+ years in product management for technical platforms, experience with LLM-based products, and strong cross-functional collaboration across engineering, ML research,

Qualifications

• 5+ years of product management experience building highly technical platform, infrastructure, or developer-focused products.
• Experience shipping production AI products powered by large language models.
• Experience defining products with LLM evaluation frameworks, AI quality measurement, and continuous model evaluation.
• Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics.

Responsibilities

• Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform.
• Build scalable LLM evaluation frameworks and automated evaluation harnesses for continuous model validation.
• Establish standardized offline and online evaluation methodologies, datasets, and quality benchmarks across AI products.
• Define the roadmap for AI observability to monitor quality, latency, cost, reliability, and user outcomes in production.
• Partner with engineering and AI researchers to enable continuous regression testing and experimentation for generative AI systems.
• Drive product decisions using AI quality metrics, customer feedback, and quantitative evaluation data.

Skills

Product management

AI products

LLM evaluation

Data-driven decisions

Communication

Cross-functional collaboration

Analytics

Tools

Datadog LLM Observability

Langfuse

LangSmith

Arize AI

Braintrust

Job description

Ridgeline is seeking a product leader to own the AI Quality Platform, partnering with engineers and researchers to define strategy and execute a roadmap for scalable evaluation, observability, and platform capabilities. You will translate AI performance signals into actionable product decisions and customer value.

The role requires 5+ years in product management for technical platforms, experience with LLM-based products, and strong cross-functional collaboration across engineering, ML research,