Applied AI Engineer — Evaluation & Reliability

15Rock; Toronto; Hybrid; CAD 110,000 - 170,000; Full time

Job summary

15Rock is an equal-opportunity employer. The Applied AI Engineer role is based in Downtown Toronto with on-site or hybrid options and authorization to work in Canada.

You will build deterministic evaluation gates, a fact-reconciliation layer, and citation integrity checks to ensure financial reports are accurate, traceable, and verifiable across tables and cited sources.

Qualifications

• Shipped LLM systems to production with high-stakes outcomes.
• Built evaluations and defined measures before testing.
• Strong Python programming for backend services and data pipelines.
• Quantitative or financial literacy to assess figures and detect errors.
• Self-directed, capable of working with minimal supervision.

Responsibilities

• Develop deterministic evaluation gates that block defective reports.
• Create a fact-reconciliation layer ensuring figures align across the report.
• Implement citation-integrity checks with live sources and references.
• Build golden-set and regression harnesses to detect quality drift.
• Hardening research into production-ready pipelines behind CI with traceability.

Skills

Python

Backend systems

Data pipelines

Quantitative literacy

Self-direction

Tools

LangGraph

DuckDB

Job description

Applied AI Engineer — Evaluation & Reliability The problem you'd own

15Rock is a financial intelligence firm. Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up. Every figure has to reconcile across every table and sentence; every claim has to trace to a live source.

We build that research with a multi-agent pipeline on large language models. LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up. Our answer isn't "prompt harder." It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work. You'd build that layer.

What you'll build

• Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel.
• A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears.
• Citation-integrity checks: every reference resolves, every quote is real, every source is live.
• Golden-set and regression harnesses that catch quality drift before it ships.
• The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works.

What we're looking for

• You've shipped LLM systems to production — not demos — where being wrong had a cost.
• You've built the evaluations that kept them honest: you think in precision and recall, and you define "good" before you measure it.
• Strong Python; you can own backend services and data pipelines yourself.
• You reason about the numbers, not just the tokens — enough quantitative or financial literacy to know when a figure is wrong, or the appetite to learn it fast.
• Self-directing. You'd rather write a one-page memo than wait for a ticket.

Bonus: model-risk or evaluation-infrastructure work in a regulated setting; LangGraph, DuckDB, CI you owned; a finance or quantitative background.

This role is not

Prompt engineering, a pure research seat, or a place where "the model said so" is an answer.

How we work

A small, senior team with direct access to the founder and research lead. Async by default — we meet on blockers, not to read status aloud. We run on hypotheses and measurable outcomes.

Details

Downtown Toronto, on-site / hybrid. Must be authorized to work in Canada. Early-stage financial intelligence firm, recognized by the Globe and Mail Changemaker Award and selected for Google for Startups.

15Rock is an equal-opportunity employer.