Remote Senior AI Evaluation & Product Quality Manager

BuddiesHR; United States; Teletrabalho; USD 150 000 - 159 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Bonus pay

Health insurance

Dental insurance

Vision insurance

401(k) plan

Paid time off

Resumo da oferta

Circana is seeking a Liquid AI Evals professional to build and own the evaluation framework for Circana's AI agents. Reporting to the Liquid AI Product Leader, you will establish repeatable testing practices that let the team move quickly while ensuring capabilities meet client, business, and technical expectations before and after launch.

You will define evaluation strategies, create test scenarios and benchmarks, implement LLM-as-a-judge methods, and monitor production traces to detect

Qualificações

• 4-6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or related field.
• Understanding of statistical evaluation, testing methodologies, and performance metrics.
• Familiarity with evaluating AI agents or LLM-based systems including LLM-as-a-judge scoring and production trace analysis.
• Ability to design structured tests for complex or ambiguous use cases.
• Strong analytical reasoning and attention to detail.
• Ability to communicate technical findings clearly to varied audiences.
• Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience.

Responsabilidades

• Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana's AI agents.
• Build evaluation frameworks for AI agent and agentic workflow capabilities.
• Partner with the Liquid AI Product Leader, engineering, and each product vertical to validate agent performance before release.
• Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods where appropriate.
• Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
• Identify failure patterns and translate findings into product improvement recommendations.
• Establish ongoing monitoring of production traces to detect performance changes following launch.
• Document evaluation results and communicate findings to technical and business audiences.
• Promote consistent evaluation standards as Liquid AI's capabilities expand across product verticals.

Conhecimentos

Product evaluation

Data science

Analytics

Experimentation

Quality assurance

Formação académica

Bachelor's degree

Descrição da oferta de emprego

Circana is seeking a Liquid AI Evals professional to build and own the evaluation framework for Circana's AI agents. Reporting to the Liquid AI Product Leader, you will establish repeatable testing practices that let the team move quickly while ensuring capabilities meet client, business, and technical expectations before and after launch.

You will define evaluation strategies, create test scenarios and benchmarks, implement LLM-as-a-judge methods, and monitor production traces to detect