Senior Product Manager - AI Evaluation & Quality (Remote)

Circana; United States; Remote; USD 150,000 - 159,000; Full time

Benefits offered by this job

Paid time off

Health insurance

401(k)

Job summary

Circana, a leading data and AI solutions provider, seeks a Liquid AI Evals professional to define evaluation frameworks and metrics for our AI agents. You’ll collaborate with engineering and product teams to validate performance before release and ensure client value through robust testing and trace analysis.

Ideal candidates have 4–6+ years in product evaluation, data science, or QA, with strong analytical and communication skills, and a degree in a quantitative field.

Qualifications

• 4-6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field.
• Understanding of statistical evaluation, testing methodologies, and performance metrics.
• Familiarity with evaluating AI agents or LLM-based systems, including LLM-as-a-judge scoring and production trace analysis.
• Ability to design structured tests for complex or ambiguous use cases.
• Strong analytical reasoning and attention to detail.
• Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience.

Responsibilities

• Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana's AI agents.
• Build evaluation frameworks for AI agent and agentic workflow capabilities.
• Partner with the Liquid AI Product Leader, engineering, and each product vertical to validate agent performance before release.
• Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods where appropriate.
• Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
• Identify failure patterns and translate findings into product improvement recommendations.
• Establish ongoing monitoring of production traces to detect performance changes following launch.
• Document evaluation results and communicate findings to technical and business audiences.
• Promote consistent evaluation standards as Liquid AI's capabilities expand across product verticals.

Skills

Product evaluation

Data science

Analytics

Experimentation

Quality assurance

Structured testing

Communication

Education

Bachelor's degree in a quantitative/technical discipline

Tools

Production tracing

LLM evaluation

Statistical testing

Job description

Circana, a leading data and AI solutions provider, seeks a Liquid AI Evals professional to define evaluation frameworks and metrics for our AI agents. You’ll collaborate with engineering and product teams to validate performance before release and ensure client value through robust testing and trace analysis.

Ideal candidates have 4–6+ years in product evaluation, data science, or QA, with strong analytical and communication skills, and a degree in a quantitative field.