Expert AI Scientist (model evaluation, DeepEval, AWS Bedrock), banking, Montreal

Abbeal; Montreal; Hybrid; CAD 110,000 - 170,000; Full time

Job summary

Abbeal Montreal is hiring an Expert AI Scientist to join the AI Factory team of a major Canadian bank. The role focuses on evaluating, hardening and improving generative AI models, with production monitoring and prompt design in a hybrid setup.

The candidate will work with AWS Bedrock and Terraform-managed infrastructure, leveraging DeepEval-like metrics and advanced tools to ensure model quality in a banking context. Bilingual French/English communication is required.

Qualifications

• Bachelor's or Master's degree in computer science, mathematics or AI
• At least 5 years in data science or AI model development
• Strong Python and AI libraries
• Hands-on Terraform and AWS
• Experience with model evaluation tools such as DeepEval
• Ability to explain technical topics simply
• Nice to have: AWS Bedrock
• Fluent French and English

Responsibilities

• Evaluate AI model performance with advanced metrics
• Fine-tune models for banking use cases
• Design and optimise prompts for generative AI applications
• Set up monitoring of models in production
• Work with the analysts and developers on AWS Bedrock and Terraform-managed infrastructure
• Keep up with the field and share best practices

Skills

Python

AI libraries

Data science

Model evaluation

Education

Bachelor's or Master's in CS/Math/AI

Tools

Terraform

AWS

AWS Bedrock

Job description

Permanent role at Abbeal Montreal: evaluate, harden and improve the generative AI models of a major Canadian bank's AI Factory. Hybrid, 2 days on site.

Abbeal is hiring an Expert AI Scientist to join the AI Factory team of a major Canadian bank in Montreal. This is a permanent position with Abbeal.

Context

The AI Factory is a team of about 25 people that takes the bank's generative AI use cases to production. Shipping a model is not enough: you need to measure how good it is, fix it and keep watching it over time. That is the core of this role.

The role

• Evaluate AI model performance with advanced metrics (DeepEval or similar)
• Fine-tune models for banking use cases
• Design and optimise prompts for generative AI applications
• Set up monitoring of models in production
• Work with the analysts and developers of the team, on AWS Bedrock and Terraform-managed infrastructure
• Keep up with the field and share best practices

What we are looking for

• Bachelor's or master's degree in computer science, mathematics or AI
• At least 5 years in data science or AI model development
• Strong Python and AI libraries
• Hands-on Terraform and AWS
• Experience with model evaluation tools such as DeepEval
• Ability to explain technical topics simply
• Nice to have: AWS Bedrock
• Fluent French and English

Terms

• Permanent position with Abbeal (employee), no freelance or subcontracting for this mandate
• Montreal, hybrid: 2 days a week in the office
• Target start: November 2026
• 12-month assignment within the AI Factory