Senior Data Scientist – AI Evaluation Platform

RBC; Toronto; On-site; CAD 150,000 - 200,000; Full time

Benefits offered by this job

Total Rewards Program

Bonuses

Stock options where applicable

Job summary

RBC is seeking a Senior Data Scientist for the AI Evaluations Platform in Toronto. You will lead evaluation design and measurement frameworks to assess model and agent quality at scale, driving evidence-based governance and production readiness.

You will design dataset construction, rubric design, human annotation protocols, and scalable evaluation methods for generative AI. Collaboration with risk, governance, and product teams is essential.

Qualifications

• 4+ years in data science or related technical field.
• Experience designing evaluation frameworks for ML/AI systems.
• Practical experience with LLM evaluation methods and human annotation protocols.
• Strong foundations in data science, statistics, ML, and programming.

Responsibilities

• Design and drive advanced model and agent evaluation methodologies (datasets, rubrics, scoring).
• Define technical standards translating risk, governance, and product quality into measurable criteria.
• Contribute to lifecycle of evaluation datasets (sourcing, curation, validation, governance).
• Design scalable evaluation methods for generative AI and agentic systems (trajectory, workflow, runtime).
• Partner with cross‑functional teams to embed evaluations into build, release, and recertification workflows.
• Establish human annotator and review protocols with reliable labels and audit-ready evidence.
• Measure scorer accuracy, calibration, robustness, and explainability.

Skills

Data science

Applied ML

Statistics

Experimental design

Programming

Tools

MLflow

Langfuse

LangSmith

OpenTelemetry

Grafana

CI/CD pipelines

Job description

RBC is seeking a Senior Data Scientist for the AI Evaluations Platform in Toronto. You will lead evaluation design and measurement frameworks to assess model and agent quality at scale, driving evidence-based governance and production readiness.

You will design dataset construction, rubric design, human annotation protocols, and scalable evaluation methods for generative AI. Collaboration with risk, governance, and product teams is essential.