Staff Data Scientist, AI Evaluations Platform

Skillhub Solutions Inc.; Montreal (administrative region); Hybride; CAD 120 000 - 180 000; Plein temps

Résumé du poste

RBC's AI Group is seeking a Staff Data Scientist to lead evaluation of model and agent quality, safety and performance. You will define standards, own datasets and scorecards, and mentor junior data scientists.

Location: Montréal, QC; Hours: 37.5 per week; Employment type: Full-time, salaried. Applications accepted until soon. You will collaborate across research, engineering, product, risk, and governance to embed evaluations into lifecycle stages.

Qualifications

• At least 8 years of experience in data science, applied ML, AI evaluation, ML quality, or a related technical field, including leadership.
• Experience designing evaluation frameworks for machine learning, generative AI, or agentic AI systems.
• Practical experience with methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination and factuality assessment, safety evaluation, or benchmarking.
• Strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML practices.
• Ability to translate governance, model risk, responsible AI, and business requirements into measurable controls and decision-ready evidence.
• Strong communication and stakeholder management skills.

Responsabilités

• Design model and agent evaluation methods, including datasets, rubrics, and measurement frameworks.
• Define standards translating model risk, responsible AI, product quality, safety, and business expectations into measurable criteria.
• Own evaluation dataset and scorecard lifecycles with sourcing, curation, validation, and lineage.
• Develop scalable evaluation methods for generative AI and agentic systems at multiple levels.
• Partner with research, engineering, product, risk, governance, and business teams to embed evaluations into development and release.
• Establish human evaluation and review protocols, and improve scorer accuracy and explainability.
• Provide technical leadership, communicate findings, and mentor junior data scientists.

Connaissances

Data science leadership

Evaluation framework design

Python

SQL

Machine learning

Stakeholder management

Formation

Master's degree or PhD in a related field

Outils

LLM-as-judge

Deterministic scoring

Human evaluation

Experimentation

Description du poste

About the opportunity

RBC's AI Group is building trusted AI capabilities for the enterprise. The Staff Data Scientist, AI Evaluations will lead the data science work for measuring model and agent quality, safety, risk, and performance. This includes developing evaluation datasets, scoring methods, human evaluation protocols, benchmarks, and measurement frameworks, while providing technical guidance and mentorship to other data scientists.

What you’ll do

• Design model and agent evaluation methods, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks.


• Define standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria and repeatable methods.


• Own evaluation dataset and scorecard lifecycles, including sourcing, curation, validation, quality checks, versioning, lineage, and ongoing improvement.


• Develop scalable evaluation methods for generative AI and agentic systems at task, workflow, trajectory, and runtime levels.


• Partner with research, engineering, product, risk, governance, and business teams to embed evaluations into development, release, certification, monitoring, and recertification.


• Establish human evaluation and review protocols, and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability.


• Provide technical leadership, communicate findings, and mentor junior data scientists.

What you’ll need

• At least 8 years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including technical leadership and mentorship.


• Experience designing evaluation frameworks for machine learning, generative AI, or agentic AI systems.


• Practical experience with methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination and factuality assessment, safety evaluation, or benchmarking.


• Strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML practices.


• Ability to translate governance, model risk, responsible AI, and business requirements into measurable controls and decision-ready evidence.


• Strong communication and stakeholder management skills.

Experience evaluating agentic systems, financial services or regulated AI, and evaluation or observability tooling is an asset. A relevant master's or Ph.D. is also an asset.

What RBC offers

A comprehensive total rewards program, development and coaching opportunities, challenging work, and a collaborative team.

Position details

Location: 6666 Rue St-Urbain, Floor 3, Montréal, Quebec
Hours: 37.5 per week
Employment type: Full-time, regular, salaried
Application deadline: Applications accepted until 11:59 PM on the day before October 31, 2026.