Data Scientist: Metrics & A/B Experiments for AI

HappyRobot; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

HappyRobot in San Francisco is seeking a data-driven analytics professional to shape product and ML roadmaps. You’ll measure how model changes, prompts, and features drive value, and turn complex data into actionable recommendations.

You will work with Product, Engineering, and ML teams to design experiments, analyze millions of conversations, and build dashboards that enable teams to understand performance and impact.

Qualifications

• 4+ years in Data Science, Product Analytics, or another highly quantitative product role.
• Strong experience with experimental design, A/B testing, statistics, causal inference, and hypothesis-driven analysis.
• Advanced proficiency in SQL and Python.
• Experience defining and operationalizing product and feature metrics.
• Ability to translate ambiguous product questions into rigorous analyses and actionable recommendations.
• Strong product instincts and the ability to distinguish statistical significance from meaningful product or customer impact.
• Experience partnering closely with Product, Engineering, or Machine Learning teams.
• Strong written and verbal communication skills.
• High attention to detail and commitment to analytical accuracy.
• Founder mindset: ownership, independence, curiosity, and willingness to go deep.

Responsibilities

• Define and track product, feature, and agent-level metrics.
• Design, run, and interpret A/B tests for model changes and product features.
• Measure how agent performance affects client outcomes and business metrics.
• Link offline model evaluations with production performance and real-world impact.
• Conduct deep analyses across conversations and workflows to identify opportunities and explain differences in performance.
• Investigate anomalies, perform root-cause analyses, and recommend improvements.
• Build statistical models, simulations, and analytical frameworks for product/ML decisions.
• Collaborate with Engineering to improve instrumentation and data quality.
• Build dashboards and self-serve tools for teams to understand performance.
• Communicate findings clearly to technical and non-technical stakeholders.

Skills

SQL

Python

Experiment design

A/B testing

Data storytelling

Stakeholder comms

Product analytics

Job description

HappyRobot in San Francisco is seeking a data-driven analytics professional to shape product and ML roadmaps. You’ll measure how model changes, prompts, and features drive value, and turn complex data into actionable recommendations.

You will work with Product, Engineering, and ML teams to design experiments, analyze millions of conversations, and build dashboards that enable teams to understand performance and impact.