Senior Product Manager, AI Evaluation & Quality| Level 18| US: Remote

Circana; United States; À distance; USD 150 000 - 159 000; Plein temps

Avantages offerts par ce poste

Bonus pay

Paid time off

Insurance (medical/dental/vision)

401(k)

Résumé du poste

Circana is seeking a Liquid AI Evals specialist to build and own the evaluation framework for AI agents. You will report to the Liquid AI Product Leader and establish repeatable testing to ensure quality, safety, and business value before and after launch.

Join a global company committed to inclusivity and continuous learning. The role requires 4–6+ years in product evaluation, data science, and performance metrics, with strong communication skills in an early-stage, fast-moving environment.

Qualifications

• 4-6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field.
• Understanding of statistical evaluation, testing methodologies, and performance metrics.
• Familiarity with evaluating AI agents or LLM-based systems, including LLM-as-a-judge scoring and production trace analysis.
• Ability to design structured tests for complex or ambiguous use cases.
• Strong analytical reasoning and attention to detail.
• Excellent communication of technical findings to varied audiences.

Responsabilités

• Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana's AI agents.
• Build evaluation frameworks for AI agent and agentic workflow capabilities.
• Partner with the Liquid AI Product Leader, engineering, and each product vertical to validate agent performance before release.
• Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods where appropriate.
• Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
• Identify failure patterns and translate findings into product improvement recommendations.
• Establish ongoing monitoring of production traces to detect performance changes following launch.
• Document evaluation results and communicate findings to technical and business audiences.
• Promote consistent evaluation standards as Liquid AI's capabilities expand across product verticals.

Connaissances

Statistical evaluation

AI agent evaluation

Structured test design

Analytical reasoning

Communication

Quality assurance

Formation

Bachelor’s degree in a quantitative, technical, or related discipline

Description du poste

If you are a job seeker with a disability and require a reasonable accommodation to apply for one of our jobs, you will find the contact information to request the appropriate accommodation by visiting the following our Accessibility Accomodation for Applicants page.

Locations

Showing 1 location

Remote- US
United States

• Remote
• Product
• Full-Time
• Requisition #: SRDIR023187

Description

Let’s be unstoppable together!

Circana is a leading provider of technology, AI, and data solutions for consumer packaged goods companies, manufacturers, and retailers. Our predictive analytics and Liquid Data® platform help clients measure market share, uncover consumer behavior, and drive growth—powered by six decades of expertise and an expansive, high-quality data set.

At Circana, we are fueled by our passion for continuous learning and growth, we seek and share feedback freely, and we celebrate victories both big and small in an environment that is flexible and accommodating to our work and personal lives. We’re a global company dedicated to fostering inclusivity and belonging. We value and celebrate the unique experiences, cultures, and viewpoints that each individual brings. By embracing a wide range of backgrounds, skills, expertise, and beyond, we create a stronger, more innovative environment for our employees, clients, and communities. With us, you can always bring your full self to work. Join our inclusive, committed team to be a challenger, own outcomes, and stay curious together. Circana is proud to be Certified™ by Great Place To Work®. This prestigious award is based entirely on what current employees say about their experience working at Circana. Learn more at www.circana.com.

Role Overview

The Liquid AI Evals role will build and own the evaluation framework used to validate the quality, reliability, safety, and business value of Circana's AI agents. Reporting into the Liquid AI Product Leader, this role will establish repeatable testing practices that let the team move quickly while ensuring new agent capabilities meet client, business, and technical expectations before and after launch.

Job Responsibilities

• Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana's AI agents.
• Build evaluation frameworks for AI agent and agentic workflow capabilities.
• Partner with the Liquid AI Product Leader, engineering, and each product vertical to validate agent performance before release.
• Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods where appropriate.
• Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
• Identify failure patterns and translate findings into product improvement recommendations.
• Establish ongoing monitoring of production traces to detect performance changes following launch.
• Document evaluation results and communicate findings to technical and business audiences.
• Promote consistent evaluation standards as Liquid AI's capabilities expand across product verticals.

Requirements

• 4-6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field.
• Understanding of statistical evaluation, testing methodologies, and performance metrics.
• Familiarity with evaluating AI agents or LLM-based systems specifically, including techniques such as LLM-as-a-judge scoring and production trace analysis, not just general machine learning models.
• Ability to design structured tests for complex or ambiguous use cases.
• Strong analytical reasoning and attention to detail.
• Comfortable operating in an early-stage, fast-moving environment with evolving requirements.
• Ability to communicate technical findings clearly to varied audiences.
• Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience.

Beyond technical skills, experience, and role-specific attributes, these shared behaviors are fundamental to our culture and success. We seek individuals who consistently demonstrate and champion these behaviors in their daily work:

• Stay Curious: Being hungry to learn and grow, always asking the big questions.
• Seek Clarity: Embracing complexity to create clarity and inspire action.
• Own the Outcome: Being accountable for decisions and taking ownership of our choices.
• Center on the Client: Relentlessly adding value for our customers.
• Be a Challenger: Never complacent, always striving for continuous improvement.
• Champion Inclusivity: Fostering trust in relationships engaging with empathy, respect, and integrity.
• Commit to each other: Contributing to making Circana a great place to work for everyone.

Location: US Remote

The below range reflects the range of possible compensation for this role at the time of this posting. We may ultimately pay more or less than the posted range. This range may be modified in the future. An employee’s position within the salary range will be based on several factors including, but not limited to, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, shift, travel requirements, sales or revenue-based metrics, any collective bargaining agreements, and business or organizational needs. The salary range for this role is $150,000 to $159,000.

This job is also eligible for bonus pay.

We offer a comprehensive package of benefits including paid time off, medical/dental/vision insurance and 401(k)to eligible employees.

An offer of employment may be conditional upon successful completion of a background check in accordance with local legislation and our candidate privacy notice . Your current employer will not be contacted without your permission.

Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities