AI Quality Engineer

Rootly

Canada

On-site

CAD 110,000 - 160,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Competitive compensation
Comprehensive medical, dental, and视s
Vacation and mental health days
Home office stipend $500
Unlimited AI tokens
Equity potential

Job summary

Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows.

You will partner with product and engineering for new AI releases, define quality metrics, and stay current on evaluation techniques. This role offers a fast-paced, high-impact environment with growth opportunities.

Qualifications

  • 5+ years in QA, product operations, AI/ML evaluation, or a closely related role.
  • Hands-on experience testing or evaluating LLM-powered or agentic AI products.
  • Strong prompt engineering instincts; understanding wording, context and structure.
  • Comfortable writing scripts or using evaluation tools; Python is a plus.
  • Sharp analytical thinking; ability to spot and articulate reasoning failures.
  • Clear written communicator to translate AI findings for technical and non-technical audiences.

Responsibilities

  • Design and execute prompt-based test scenarios across agentic AI features.
  • Evaluate AI outputs for accuracy, relevance, and workflow alignment.
  • Build and maintain an evaluation framework, scoring rubrics, and regression suites.
  • Identify failure modes and document findings with engineers to resolve them.
  • Partner with Product and Engineering on new AI feature releases and gates.
  • Define and track quality metrics and report findings to stakeholders.
  • Stay current on LLM evaluation, prompt engineering, and testing methodologies.

Skills

QA experience
LLM evaluation
Prompt engineering
Scripting (Python)
Analytical thinking
Clear communication

Tools

Playwright
LangSmith
PromptFlow
Braintrust

Job description

About Rootly

At Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving out a new multi-billion-dollar category ourselves and need incredible talent to achieve this ambitious goal together.

Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.

Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.

Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.

This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.

What you'll do
  • Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
  • Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
  • Build and maintain an evaluation framework; structured test libraries, scoring rubrics, and regression suites to track AI performance over time
  • Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours; document findings and work with engineers to resolve them
  • Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
  • Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
  • Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
  • +5 years in QA, product operations, AI/ML evaluation, or a closely related role
  • Hands-on experience testing or evaluating LLM-powered or agentic AI products
  • Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
  • Comfortable writing scripts or working with evaluation tools(Python a plus; not required to be a full-stack engineer)
  • Sharp analytical thinking; you can spot a subtle reasoning failure and articulate exactly why it's a problem
  • Clear written communicator; able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
  • Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
  • Experience with evaluation frameworks (e.g. LangSmith, PromptFlow, Braintrust, or similar)
  • Exposure to red‑teaming or adversarial testing of AI systems
  • Comfortable writing E2E tests with Playwright
  • Background working at a B2B SaaS or developer‑tools company
  • Familiar with mobile app testing (iOS/Android)
Why Rootly?

We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.

Here's what you can expect when you join Rootly:

  • Competitive compensation and early equity in a fast‑growing, venture‑backed company.
  • Comprehensive medical, dental, and vision coverage.
  • 3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
  • $500 stipend for home office setup.
  • Unlimited token usage and access to AI tools
  • A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.

This is a newly created vacancy.

As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.

Accommodations are available on request at any stage of our recruitment process. Contact recruitment@rootly.com and we will work with you to meet your needs confidentially.

Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer
Senior AI Engineer

Rootly • Toronto

On-site
CAD 120,000 - 180,000
Equity in a fast-growing company
Comprehensive medical/dental/vision
Three weeks of vacation + sick days
+2
Senior Platform Engineer, Developer Experience
Senior Platform Engineer, Developer Experience

Rootly • Toronto

On-site
CAD 120,000 - 180,000
Equity
Medical coverage
Vacation policy
+3
Platform Engineer, Infrastructure
Platform Engineer, Infrastructure

Rootly • Toronto

On-site
CAD 120,000 - 170,000
Competitive compensation & equity
Comprehensive medical, dental, and vis
3 weeks vacation + sick days
+2
Staff Platform Engineer
Staff Platform Engineer

Rootly • Canada

On-site
CAD 140,000 - 210,000
Competitive compensation and early-equ
Equity in a fast-growing company
Comprehensive medical, dental, and vis
+4
Member of Technical Staff, Intern
Member of Technical Staff, Intern

Rootly • Toronto

Hybrid
CAD 24,000 - 36,000
Senior Platform Engineer, Datastores
Senior Platform Engineer, Datastores

Rootly • Toronto

On-site
CAD 140,000 - 190,000
Competitive compensation
Early equity
Medical, dental, vision coverage
+2
Senior Quality Assurance Specialist (AI Testing / QA Lead)
Senior Quality Assurance Specialist (AI Testing / QA Lead)

Altis Technology • Toronto

Hybrid
CAD 96,000 - 138,000
Hybrid schedule
Solutions Engineer
Solutions Engineer

rootly • Toronto

On-site
CAD 100,000 - 150,000
Equity
Medical/Dental/Vision
Vacation + sick leave
+2
Senior QA Engineer Toronto
Senior QA Engineer Toronto

Konrad • Toronto

Hybrid
CAD 100,000 - 125,000
Retirement Planning
Parental Leave Program
Annual tech & travel allowance
+5
Senior Security Engineer, Enterprise Security
Senior Security Engineer, Enterprise Security

Rootly • Toronto

On-site
CAD 120,000 - 180,000
Competitive compensation
Equity for early employees
Health, dental, vision coverage
+2