Agentic AI QA Engineer

XYZ Venture Capital; Toronto; On-site; CAD 90,000 - 130,000; Full time

Benefits offered by this job

Equity (venture-backed)

Comprehensive medical/dental/vision

3 weeks vacation + unlimited sick/well

Home office stipend $500

Unlimited AI tools access

Job summary

Rootly is seeking an AI Quality Engineer to own the evaluation and optimization of our agentic AI features—designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.

This role involves thinking like an attacker, a curious user, and a power user across complex incident workflows to probe how our AI agents reason, decide, and handle edge cases.

Qualifications

• 5+ years in QA, product operations, AI/ML evaluation or related roles.
• Hands-on experience testing or evaluating LLM-powered or agentic AI products.
• Strong prompt engineering instincts—understand how wording and context affect model behaviour.
• Comfortable writing scripts or using evaluation tools (Python a plus).
• Sharp analytical thinking with ability to articulate reasoning failures.
• Clear written communicator for technical and non-technical audiences.

Responsibilities

• Design and execute prompt-based test scenarios for happy paths, edge cases, and adversarial inputs.
• Evaluate AI outputs for accuracy, relevance, and alignment with workflows.
• Build and maintain an evaluation framework with test libraries, rubrics, and regression suites.
• Identify failure modes, hallucinations, and reasoning gaps; document findings with engineers.
• Partner with Product and Engineering on new AI feature releases and quality gates.
• Define and track quality metrics (accuracy, failure frequency, regression trends).
• Stay current on LLM evaluation techniques, prompt engineering practices, and agentic testing methodologies.

Skills

5+ years QA/AI evaluation

LLM/agentic AI testing

Prompt engineering

Scripting (Python)

Analytical thinking

Clear written communication

Tools

Python

Job description

Rootly is seeking an AI Quality Engineer to own the evaluation and optimization of our agentic AI features—designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.

This role involves thinking like an attacker, a curious user, and a power user across complex incident workflows to probe how our AI agents reason, decide, and handle edge cases.