AI Product Manager - LLM Systems & Data Quality

CENTRL; Mountain View (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

CENTRL is seeking a product-oriented AI Quality Owner for CentrlX, focusing on accuracy, reliability, and cost efficiency of the AI behind our agentic platform. You will manage evaluation datasets, run cross-model comparisons, and drive improvements into the product while collaborating with Sales and Professional Services.

This is a product role blending data science rigor with product judgment to ship impactful changes and define acceptance criteria for prompts, models, and configurations.

Qualifications

• 3+ years across product management, data science, or applied AI, including at least 2 years working on LLM-based products in production.

Responsibilities

• Own AI Quality and be the point person for what is happening and what needs to improve.
• Define what 'good' means for each core workflow (digitization, retrieval, summarization, evaluation) and set measurable quality bars.
• Benchmark models across providers on accuracy, latency, and cost for each workflow; own migrations and cost optimization.
• Define logging and tracing, and partner with product/design to capture in-app feedback to accumulate labeled data.

Skills

Python

SQL

LLM evaluation

A/B testing

Data labeling

Documentation

Communication

Cost optimization

Agile

Education

Bachelor's degree in a quantitative or technical field

Tools

Braintrust

LangSmith

Langfuse

Arize

Weave

Job description

CENTRL is seeking a product-oriented AI Quality Owner for CentrlX, focusing on accuracy, reliability, and cost efficiency of the AI behind our agentic platform. You will manage evaluation datasets, run cross-model comparisons, and drive improvements into the product while collaborating with Sales and Professional Services.

This is a product role blending data science rigor with product judgment to ship impactful changes and define acceptance criteria for prompts, models, and configurations.