AI Quality Platform Product Manager

Ridgeline; Reno (NV); On-site; USD 136,000 - 187,000; Full time

Benefits offered by this job

Unlimited vacation

Educational reimbursements

Wellness reimbursements

Zero-cost employee insurance

Stock options

Job summary

Qualifications

• 5+ years of product management experience building highly technical platform, infra, or developer-focused products.
• Experience shipping production AI products powered by large language models.
• Experience defining products with LLM evaluation frameworks, AI quality measurement, and continuous model evaluation.
• Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics.
• Experience with LLM observability platforms and related AI quality tools.

Responsibilities

• Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform.
• Build scalable LLM evaluation frameworks and automated evaluation harnesses for continuous model validation.
• Establish standardized offline and online evaluation methodologies and quality benchmarks.
• Define the roadmap for AI observability to monitor quality, latency, cost, and reliability in production.
• Partner with engineering and AI researchers to enable continuous regression testing and experimentation.
• Drive product decisions using AI quality metrics, feedback, and data.
• Collaborate to deliver platform capabilities for trusted, measurable AI experiences.
• Translate advances in foundation models into platform capabilities that create customer value.
• Prioritize investments using insights and measurable business outcomes.
• Communicate vision and priorities clearly across technical and executive stakeholders.
• Foster ownership, learning, experimentation, and transparency.

Skills

Product management

AI platform

LLM evaluation

Observability

Data analysis

Stakeholder communication

Tools

Datadog LLM Observability

Langfuse

LangSmith

Arize AI

Braintrust

Galileo

Job description