Lead Data Scientist, AI Evaluation & Quality Benchmarks

Skillhub Solutions Inc.; Montreal (administrative region); Hybrid; CAD 120,000 - 180,000; Full time

Job summary

RBC's AI Group is seeking a Staff Data Scientist to lead evaluation of model and agent quality, safety and performance. You will define standards, own datasets and scorecards, and mentor junior data scientists.

Location: Montréal, QC; Hours: 37.5 per week; Employment type: Full-time, salaried. Applications accepted until soon. You will collaborate across research, engineering, product, risk, and governance to embed evaluations into lifecycle stages.

Qualifications

• At least 8 years of experience in data science, applied ML, AI evaluation, ML quality, or a related technical field, including leadership.
• Experience designing evaluation frameworks for machine learning, generative AI, or agentic AI systems.
• Practical experience with methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination and factuality assessment, safety evaluation, or benchmarking.
• Strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML practices.
• Ability to translate governance, model risk, responsible AI, and business requirements into measurable controls and decision-ready evidence.
• Strong communication and stakeholder management skills.

Responsibilities

• Design model and agent evaluation methods, including datasets, rubrics, and measurement frameworks.
• Define standards translating model risk, responsible AI, product quality, safety, and business expectations into measurable criteria.
• Own evaluation dataset and scorecard lifecycles with sourcing, curation, validation, and lineage.
• Develop scalable evaluation methods for generative AI and agentic systems at multiple levels.
• Partner with research, engineering, product, risk, governance, and business teams to embed evaluations into development and release.
• Establish human evaluation and review protocols, and improve scorer accuracy and explainability.
• Provide technical leadership, communicate findings, and mentor junior data scientists.

Skills

Data science leadership

Evaluation framework design

Python

SQL

Machine learning

Stakeholder management

Education

Master's degree or PhD in a related field

Tools

LLM-as-judge

Deterministic scoring

Human evaluation

Experimentation

Job description

RBC's AI Group is seeking a Staff Data Scientist to lead evaluation of model and agent quality, safety and performance. You will define standards, own datasets and scorecards, and mentor junior data scientists.

Location: Montréal, QC; Hours: 37.5 per week; Employment type: Full-time, salaried. Applications accepted until soon. You will collaborate across research, engineering, product, risk, and governance to embed evaluations into lifecycle stages.