AI Observability Engineer — Build & Govern Production LLMs (Hybrid)

BMO; Toronto; Hybrid; CAD 80,000 - 175,000; Full time

Benefits offered by this job

Health insurance

Tuition reimbursement

Accident and life insurance

Retirement savings plans

Job summary

BMO Financial Group is seeking a Hybrid Senior AI Observability Engineer in Canada to design, build, and operate systems that track and evaluate non-deterministic AI models and agent workflows. You will trace LLM calls, monitor token usage, detect drift, and run automated evaluations in production.

You should have 7–10 years of Site Reliability Engineering experience, strong automation and observability skills, and experience with Python, SQL, and telemetry tools (Prometheus, Grafana, Datadog).

Qualifications

• Bachelor's degree in Computer Science, Engineering or a related domain.
• 7–10 years experience in Site Reliability Engineering.
• Experience in AI Observability Engineering to automate detect/investigate/remediate.
• Exposure to Platform Engineering and building AI capabilities.
• Strong automation, monitoring and observability skills; understands model architecture, evaluation metrics, token usage and drift.
• Proficiency in Python for evaluations and SQL for telemetry queries.

Responsibilities

• Designs, builds and maintains systems that track, evaluate and debug AI models and agent workflows.
• Trace LLM calls, monitor token usage, detect drift, and set up automated evaluations.
• Govern autonomous agent behaviors in production and run ML tests/experiments.
• Train and retrain systems to prevent drift and optimize results.
• Provision, configure and standardize observability pipelines and dashboards with Prometheus, Grafana and Datadog.

Skills

Site Reliability Engineering

AI Observability

Python

SQL

Prometheus

Grafana

Datadog

Education

Bachelor's Degree

Tools

Prometheus

Grafana

Datadog

Job description

BMO Financial Group is seeking a Hybrid Senior AI Observability Engineer in Canada to design, build, and operate systems that track and evaluate non-deterministic AI models and agent workflows. You will trace LLM calls, monitor token usage, detect drift, and run automated evaluations in production.

You should have 7–10 years of Site Reliability Engineering experience, strong automation and observability skills, and experience with Python, SQL, and telemetry tools (Prometheus, Grafana, Datadog).