Senior AI Product Analytics Lead

Angi; United States; Remote; USD 165,000 - 225,000; Full time

Benefits offered by this job

Year-end bonus

Equity

Medical benefits

Flexible vacation

401K match

Job summary

Angi is seeking a Staff AI Product Analyst to define AI-native product analytics, aligning AI capabilities with practical business outcomes. You will shape evaluation frameworks, datasets, and governance while influencing product analytics across teams.

You will lead flagship AI journeys, partner with AI product, engineering, and data teams, and translate complex AI concepts into actionable business insights, all while ensuring trust and quality in data-driven decisions.

Qualifications

• 10+ years of experience across product analytics, data science, or technical product management.
• 1+ years of hands-on experience working with AI/LLM systems in an applied context, focusing on evaluating, trusting, and deploying them responsibly.
• Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices.
• Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline.
• Exceptional communication skills with the ability to translate highly technical AI insights for non-technical business stakeholders.

Responsibilities

• Flagship Journey Measurement: define north-star and guardrail metrics for flagship AI journeys focusing on adoption, task completion, quality, and outcomes.
• Causal Framing & Experimentation: align with data science on test design, interpretation, and causal framing when new AI products ship.
• Data Layer Systems Thinking: map how agents interact with metrics, raw tables, documentation, and APIs to ensure data/analytics requirements are explicit and testable.
• Eval Framework Architecture: design and maintain evaluation sets, golden datasets, rubrics, and regression checks for agent outputs to ensure accuracy, safety, and usefulness.
• Taxonomy & Failure Mode Analysis: categorize wrong answers, tool misuse, context gaps, and data edge cases to feed insights back into prompts, retrieval, tools, and data pipelines.
• Performance Monitoring: monitor live usage of agentic AI analytics workflows and track success, abandonment, fallback patterns, and human overrides to prioritize improvements.
• Playbooks & Narrative: own internal docs defining when to use agents, human checkpoints, and uncertainty communication; support enablement and leadership storytelling.
• Strategic Collaboration: partner with business leaders, analytics engineers, and data teams to translate AI capabilities into measurable outcomes.
• Defining Success: deliver recurring measurement artifacts used by leadership to prioritize roadmaps and reduce high-severity incidents where metrics are incorrect.

Skills

Product analytics

Data science

AI evaluation

Python

Communication skills

Tools

ML frameworks

Data visualization

Data governance

Analytics platforms

Job description

Angi is seeking a Staff AI Product Analyst to define AI-native product analytics, aligning AI capabilities with practical business outcomes. You will shape evaluation frameworks, datasets, and governance while influencing product analytics across teams.

You will lead flagship AI journeys, partner with AI product, engineering, and data teams, and translate complex AI concepts into actionable business insights, all while ensuring trust and quality in data-driven decisions.