Applied AI Engineer — Evaluation & Reliability

15Rock, Ltd.; Toronto; Hybrid; CAD 110,000 - 170,000; Full time

Benefits offered by this job

Competitive compensation

Hybrid work model

Job summary

15Rock, Ltd. is hiring an Applied AI Engineer — Evaluation & Reliability in Canada to build deterministic evaluation gates, fact-reconciliation, and citation integrity for financial intelligence reports.

You will work on multi-agent LLM pipelines and own backend data pipelines in a small, senior team. The role emphasizes precision over novelty, shipping production systems, and using Python, LangGraph, and DuckDB.

Qualifications

• Shipped LLM systems to production with measurable impact.
• Built evaluation gates and fact-checking pipelines.
• Strong Python and data-pipeline engineering skills.
• Finance/quantitative literacy to spot wrong figures and learn fast.

Responsibilities

• Build deterministic evaluation gates that block a report when defects are found.
• Create a fact-reconciliation layer ensuring consistency of figures across reports.
• Implement citation-integrity checks with live sources and verifiable quotes.
• Develop golden-set and regression harnesses to catch quality drift before release.

Skills

Python

Data pipelines

Backend services

Quantitative literacy

Tools

LangGraph

DuckDB

CI/CD

Job description

Applied AI Engineer — Evaluation & Reliability

15Rock is a financial intelligence firm. Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up. Every figure has to reconcile across every table and sentence; every claim has to trace to a live source.

We build that research with a multi-agent pipeline on large language models. LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up. Our answer isn't "prompt harder." It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work. You'd build that layer.

What you'll build

• Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel.
• A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears.
• Citation-integrity checks: every reference resolves, every quote is real, every source is live.
• Golden-set and regression harnesses that catch quality drift before it ships.
• The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works.

What we're looking for

• You've shipped LLM systems to production — not demos — where being wrong had a cost.
• You've built the evaluations that kept them honest: you think in precision and recall, and you define "good" before you measure it.
• Strong Python; you can own backend services and data pipelines yourself.
• You reason about the numbers, not just the tokens — enough quantitative or financial literacy to know when a figure is wrong, or the appetite to learn it fast.
• Self-directing. You'd rather write a one-page memo than wait for a ticket.

Bonus: model-risk or evaluation-infrastructure work in a regulated setting; LangGraph, DuckDB, CI you owned; a finance or quantitative background.

This role is not

Prompt engineering, a pure research seat, or a place where "the model said so" is an answer.

How we work

A small, senior team with direct access to the founder and research lead. Async by default — we meet on blockers, not to read status aloud. We run on hypotheses and measurable outcomes.

Details

Downtown Toronto, on-site / hybrid. Must be authorized to work in Canada. Early-stage financial intelligence firm, recognized by the Globe and Mail Changemaker Award and selected for Google for Startups.

15Rock is an equal-opportunity employer.