Senior AI Evaluation & Quality Product Lead

Circana; United States; Remote; USD 150,000 - 159,000; Full time

Benefits offered by this job

Bonus eligibility

401(k) plan

Medical/dental/vision insurance

Job summary

Circana is seeking a Senior Product Manager for AI Evaluation & Quality to design and own evaluation frameworks for Circana's AI agents. You will align with the Liquid AI Product Leader, engineers, and product verticals to ensure performance and value before and after launch.

You will develop test scenarios, golden datasets, and scoring approaches, including LLM-as-a-judge methods, and monitor production traces to detect shifts in performance while communicating results to technical and business

Qualifications

• 4–6+ years in product evaluation, data science, analytics, or QA.
• Experience evaluating AI agents/LLM-based systems.
• Ability to design structured tests for complex use cases.
• Strong analytical reasoning and communication skills.

Responsibilities

• Define evaluation strategies, methodologies, metrics, and thresholds.
• Build evaluation frameworks for AI agent and workflow capabilities.
• Partner with Liquid AI Product Leader, engineering, and product verticals.
• Develop test scenarios, golden datasets, benchmarks, and scoring approaches.
• Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
• Identify failure patterns and translate findings into improvements.
• Establish monitoring of production traces post-launch.
• Document results and communicate findings to technical/business audiences.
• Promote consistent evaluation standards as capabilities expand.

Skills

Statistical evaluation

Experimentation

Data analysis

Strong analytical reasoning

Clear technical communication

Adaptability in fast-moving env

Education

Bachelor’s degree in quantitative/technical field

Job description

Circana is seeking a Senior Product Manager for AI Evaluation & Quality to design and own evaluation frameworks for Circana's AI agents. You will align with the Liquid AI Product Leader, engineers, and product verticals to ensure performance and value before and after launch.

You will develop test scenarios, golden datasets, and scoring approaches, including LLM-as-a-judge methods, and monitor production traces to detect shifts in performance while communicating results to technical and business