AI Quality Engineer: Prompt Testing & Evaluation

Rootly

Canada

On-site

CAD 110,000 - 160,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Competitive compensation
Comprehensive medical, dental, and视s
Vacation and mental health days
Home office stipend $500
Unlimited AI tokens
Equity potential

Job summary

Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows.

You will partner with product and engineering for new AI releases, define quality metrics, and stay current on evaluation techniques. This role offers a fast-paced, high-impact environment with growth opportunities.

Qualifications

  • 5+ years in QA, product operations, AI/ML evaluation, or a closely related role.
  • Hands-on experience testing or evaluating LLM-powered or agentic AI products.
  • Strong prompt engineering instincts; understanding wording, context and structure.
  • Comfortable writing scripts or using evaluation tools; Python is a plus.
  • Sharp analytical thinking; ability to spot and articulate reasoning failures.
  • Clear written communicator to translate AI findings for technical and non-technical audiences.

Responsibilities

  • Design and execute prompt-based test scenarios across agentic AI features.
  • Evaluate AI outputs for accuracy, relevance, and workflow alignment.
  • Build and maintain an evaluation framework, scoring rubrics, and regression suites.
  • Identify failure modes and document findings with engineers to resolve them.
  • Partner with Product and Engineering on new AI feature releases and gates.
  • Define and track quality metrics and report findings to stakeholders.
  • Stay current on LLM evaluation, prompt engineering, and testing methodologies.

Skills

QA experience
LLM evaluation
Prompt engineering
Scripting (Python)
Analytical thinking
Clear communication

Tools

Playwright
LangSmith
PromptFlow
Braintrust

Job description

Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows.

You will partner with product and engineering for new AI releases, define quality metrics, and stay current on evaluation techniques. This role offers a fast-paced, high-impact environment with growth opportunities.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality Engineer
AI Quality Engineer

Centraprise • Vancouver

On-site
CAD 80,000 - 100,000
AI Quality Engineer
AI Quality Engineer

Rootly • Canada

On-site
CAD 110,000 - 160,000
Competitive compensation
Comprehensive medical, dental, and视s
Vacation and mental health days
+3
Senior AI Test Automation Engineer
Senior AI Test Automation Engineer

PureFacts Financial Solutions • Toronto

On-site
CAD 100,000 - 140,000
Senior Quality Engineer – Test Automation, AI Testing & Java/Python
Senior Quality Engineer – Test Automation, AI Testing & Java/Python

Astra-North Infoteck Inc. • Toronto

On-site
CAD 90,000 - 120,000
Staff Quality Engineer - AI Systems & Automation
Staff Quality Engineer - AI Systems & Automation

Devconnectplatform • Canada

Remote
CAD 134,000 - 185,000
Health benefits from day one
Remote-first
Generous PTO
+1
QA Engineer: AI-Driven Test Automation
QA Engineer: AI-Driven Test Automation

Method Integration Inc. • Toronto

Hybrid
CAD 50,000 - 60,000
Hybrid work model
Weekly lunch credit
Health benefits
+1
Quality Engineer (QE) / SDET – AI Chatbot Validation
Quality Engineer (QE) / SDET – AI Chatbot Validation

Astra-North Infoteck Inc. • Toronto

On-site
CAD 110,000 - 160,000
Quality Engineer (QE) / SDET – AI Chatbot Validation
Quality Engineer (QE) / SDET – AI Chatbot Validation

Astra-North Infoteck Inc. • Toronto

On-site
CAD 90,000 - 140,000
Quality Assurance Analyst
Quality Assurance Analyst

GuestTek Interactive Entertainment Ltd. • Calgary

On-site
CAD 70,000 - 90,000
QA Automation Engineer - Remote, AI-Powered Testing
QA Automation Engineer - Remote, AI-Powered Testing

360training.com • Canada

Remote
CAD 90,000 - 120,000
Monthly Team Events
Pet Insurance
Remote & Hybrid Work Models
+3