Senior Data Scientist, AI Product Performance

Microsoft; Vancouver; On-site; CAD 114,000 - 204,000; Full time

Job summary

A leading technology company in Vancouver seeks a Data Science professional to establish metrics for AI products, implement robust data pipelines, and deliver actionable insights. Candidates should have significant experience in data science, particularly with SQL and Python, and hold a relevant doctorate, master's, or bachelor's degree. The typical base pay range for this role across Canada is CAD $114,400 - CAD $203,900 annually.

Qualifications

• 1+ years of data-science experience for Doctorate holders.
• 3+ years of data-science experience for Master's Degree holders.
• 5+ years of data-science experience for Bachelor's Degree holders.

Responsibilities

• Establish evaluation metrics for AI products.
• Implement data pipelines using Python, SQL/KQL, and Power BI.
• Deliver insights from user engagement analytics.
• Analyze conversational product interactions for user patterns.
• Develop scalable analytics functions and automated data workflows.

Skills

Data analysis

Machine learning

Python

SQL

Data visualization

Education

Doctorate in Data Science, Mathematics, or related field

Master's Degree in related field

Bachelor's Degree in related field

Tools

Power BI

Azure Data Explorer

Kafka

Job description

Responsibilities

• Establish standardized scorecard evaluation metrics for AI products, covering user engagement, model performance, and product quality, to enable consistent tracking across teams and disciplines.
• Implement robust data pipelines and evaluation calculations using tools such as Python, C#, SQL/KQL, Power BI, and LLM prompting techniques; collaborate with engineering teams to ensure solutions are scalable, reliable, and integrated into product workflows for real-time and offline testing.
• Deliver actionable insights by analyzing analytics to measure user acquisition, activation, engagement, retention, churn, and ecosystem health; identify key patterns and feedback loops to inform product and model improvements.
• Analyze conversational product interactions—language content, semantics, dialogue structure—to uncover user intent and behavior; utilize topic clustering, semantic embedding, and build lightweight classifiers for usage pattern modeling.
• Present and communicate product data insights through dashboards, reports, and stakeholder discussions, enabling understanding of user behavior and performance through qualitative and quantitative analysis.
• Develop scalable, maintainable pipelines and reusable analytics functions from ad hoc analyses; automate data workflows and support ML experimentation with high-quality datasets.
• Apply offline and online evaluation methods—model regression tests, A/B testing, real-world usage tracking—and deploy classifiers/scoring models to evaluate AI output quality (relevance, helpfulness, coherence) at scale.

Qualifications

Required Qualifications (RQs)

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ years of data-science experience.
• OR Master's Degree in related fields AND 3+ years of data-science experience.
• OR Bachelor's Degree in related fields AND 5+ years of data-science experience.
• OR equivalent experience.
• 3+ years supporting a consumer product and guiding product roadmaps with data.
• Proficiency in SQL and Python (or R), with experience in common data science libraries.
• Experience with real-time data systems, large-scale product data (e.g., Kafka, Azure Data Explorer), ML infrastructure, and building lightweight classifiers or recommendation components.

Preferred Qualifications (PQs)

• Ability to work in ambiguous problem spaces and design analytical frameworks from scratch.
• Experience supporting or building evaluation systems for language models, recommendation engines, or personalization systems.
• Experience with analytics in digital consumer products, especially AI/LLM features, including experiment design and causal inference analysis.

Data Science IC4 - The typical base pay range for this role across Canada is CAD $114,400 - CAD $203,900 per year. Find additional pay information here. Applications accepted until August 11, 2025. Microsoft is an equal opportunity employer, considering all qualified applicants without discrimination. For assistance or accommodations due to disability, please send a request via the Accommodation request form.