Senior AI Engineer for Observability

BMO; Toronto; Hybrid; CAD 80,000 - 175,000; Full time

Benefits offered by this job

Health insurance

Tuition reimbursement

Accident and life insurance

Retirement savings plans

Job summary

BMO Financial Group is seeking a Hybrid Senior AI Observability Engineer in Canada to design, build, and operate systems that track and evaluate non-deterministic AI models and agent workflows. You will trace LLM calls, monitor token usage, detect drift, and run automated evaluations in production.

You should have 7–10 years of Site Reliability Engineering experience, strong automation and observability skills, and experience with Python, SQL, and telemetry tools (Prometheus, Grafana, Datadog).

Qualifications

• Bachelor's degree in Computer Science, Engineering or a related domain.
• 7–10 years experience in Site Reliability Engineering.
• Experience in AI Observability Engineering to automate detect/investigate/remediate.
• Exposure to Platform Engineering and building AI capabilities.
• Strong automation, monitoring and observability skills; understands model architecture, evaluation metrics, token usage and drift.
• Proficiency in Python for evaluations and SQL for telemetry queries.

Responsibilities

• Designs, builds and maintains systems that track, evaluate and debug AI models and agent workflows.
• Trace LLM calls, monitor token usage, detect drift, and set up automated evaluations.
• Govern autonomous agent behaviors in production and run ML tests/experiments.
• Train and retrain systems to prevent drift and optimize results.
• Provision, configure and standardize observability pipelines and dashboards with Prometheus, Grafana and Datadog.

Skills

Site Reliability Engineering

AI Observability

Python

SQL

Prometheus

Grafana

Datadog

Education

Bachelor's Degree

Tools

Prometheus

Grafana

Datadog

Job description

Address: 33 Dundas Street West

Job Family Group: Data Analytics & Reporting

Hybrid role

Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production. Designs and develops machine learning (ML) and deep learning systems. Runs machine learning tests and experiments. Trains and retrain systems to prevent drift and optimize results.

Qualifications:

• Bachelor's Degree in Computer Science, Engineering or other relevant domain
• 7-10 years experience in Site Reliability Engineering
• Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
• Exposure to Platform Engineering, building AI capabilities
• Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift
• Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
• Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
• Exposure to heavily regulated industry organizations

Agent & LLM Tracing

• Implement OpenTelemetry-based frameworks to capture step-by-step agent trajectories, spans, prompts, tool calls, and intermediate outputs.
• Design automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence using LLM-as-a-judge or custom code.
• Track model drift, data quality degradation, latency spikes, and error rates across production pipelines.
• Build telemetry and sampling strategies to control high data and inference costs, managing log retention and cardinality.
• Define operational boundaries for autonomous AI agents, setting rules for when they can self-remediate.
• Reduce alert fatigue by replacing static thresholds with dynamic, intelligent anomaly detection and building runbooks for AI failures.

Salary

Salary: $80,000.00 - $175,000.00 Pay Type: Salaried. The above represents BMO Financial Group’s pay range and type. Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part‑time roles will be pro‑rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position. BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards.

BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans.

To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/Total-Rewards

About Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world. As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset. To find out more visit us at https://jobs.bmo.com/ca/en.

BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.

BMO is a leading bank driven by a single purpose: to Boldly Grow the Good in business and life. Everywhere we do business, we’re focused on building, investing and transforming how we work to drive performance and continue growing the good. Who we are We’re proud to be fueling growth and expanding possibilities for individuals, families and businesses. More than 12 million customers count on us for personal and commercial banking, wealth management and investment services. As the 8th largest bank in North America by assets, we provide personal and commercial banking, wealth management and investment services to more than 12 million customers. In Canada, the United States and across the globe, we’ll continue to build, invest and transform to drive performance that serves the good that grows.