Senior Data Scientist, AI Evaluations Platform

RBC

Toronto

On-site

CAD 150,000 - 200,000

Full time

4 days ago

Be an early applicant

Application generator

Stand out for this role — generate a tailored resume and cover letter in about a minute.

Get past ATS filters

Benefits offered by this job

Total Rewards Program

Bonuses

Stock options where applicable

Job summary

RBC is seeking a Senior Data Scientist for the AI Evaluations Platform in Toronto. You will lead evaluation design and measurement frameworks to assess model and agent quality at scale, driving evidence-based governance and production readiness.

You will design dataset construction, rubric design, human annotation protocols, and scalable evaluation methods for generative AI. Collaboration with risk, governance, and product teams is essential.

Qualifications

4+ years in data science or related technical field.
Experience designing evaluation frameworks for ML/AI systems.
Practical experience with LLM evaluation methods and human annotation protocols.
Strong foundations in data science, statistics, ML, and programming.

Responsibilities

Design and drive advanced model and agent evaluation methodologies (datasets, rubrics, scoring).
Define technical standards translating risk, governance, and product quality into measurable criteria.
Contribute to lifecycle of evaluation datasets (sourcing, curation, validation, governance).
Design scalable evaluation methods for generative AI and agentic systems (trajectory, workflow, runtime).
Partner with cross‑functional teams to embed evaluations into build, release, and recertification workflows.
Establish human annotator and review protocols with reliable labels and audit-ready evidence.
Measure scorer accuracy, calibration, robustness, and explainability.

Skills

Data science

Applied ML

Statistics

Experimental design

Programming

Tools

MLflow

Langfuse

LangSmith

OpenTelemetry

Grafana

CI/CD pipelines

Job description

Senior Data Scientist, AI Evaluations Platform

Sep 2, 2026

Job Description

What is the opportunity?

RBC's AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of the core controls that makes that possible. As Senior Data Scientist, AI Evaluations Platform, you will be a technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk, and performance at scale. You will design, build, and continuously improve evaluations: dataset sourcing and curation; scorer design, testing, and benchmarking; human evaluation protocols; and measurement frameworks that help AI systems move from experimentation to production with evidence and control.

What will you do?

Design and drive advanced model and agent evaluation methodologies, including dataset construction, rubric design, scoring methodologies and meta-evaluation, human annotation protocols, and measurement frameworks.

Define technical standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.

Contribute to the end-to-end lifecycle for evaluation datasets, including sourcing, curation, validation, quality checks, governance, reuse, and ongoing improvement.

Design scalable evaluation methodologies for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.

Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.

Establish formal human annotator and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.

Measure and improve scorer accuracy, calibration, robustness, failure mode coverage, and explainability.

What do you need to succeed?

Must Have

4+ years of experience in data science, applied machine learning, computational mathematics, model evaluation, or a related technical field.

Experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and measurements.

Practical experience with LLM evaluation methods such as LLM-as-a-Judge, deterministic scoring, and human annotation protocols.

Strong technical foundations in data science, statistics, machine learning, experimental design, and programming.

Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.

Nice to Have

Experience evaluating agentic AI systems, tool-calling workflows, multi-step reasoning, runtime traces, trajectory scoring, or workflow-level performance.

Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit-ready evidence processes.

Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.

Publications, patents, open-source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.

What's in it for you?

We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.

A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable.

Leaders who support your development through coaching and managing opportunities.

Ability to make a difference and lasting impact.

Work in a dynamic, collaborative, progressive, and high-performing team.

A world-class training program in financial services.

Opportunities to do challenging work.

#LI-Post

#TECHPJ

Job Skills

AI Systems, Applied Research, Big Data Analytics, Critical Thinking, Data Science, Decision Making, Machine Learning (ML), Model Evaluation, Software Engineering, Statistics

Additional Job Details

Address: RBC WATERPARK PLACE, 88 QUEENS QUAY W:TORONTO

City: Toronto

Country: Canada

Work hours/week: 37.5

Employment Type: Full time

Platform: TECHNOLOGY AND OPERATIONS

Job Type: Regular

Pay Type:

Posted Date: 2026-09-02

Application Deadline: 2026-09-20

Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above

Company size 10,001+ employees

Company type Public company

Total funding $22.2M Post IPO Debt

Momentum

At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.

Team growth

Momentum

6% in 12 mo

98,739 employees on LinkedIn

Feb 23 May 26

Employee experience

What it's like inside

3.8

19,829 reviews

79% would recommend

Culture & values

Culture & values 3.9