Senior AI QA Engineer – AI Evaluation Lead

Lever, Inc.; Canada; Remote; CAD 100,000 - 130,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401k with company match

Life Insurance

Mental health benefits

Job summary

Novara is seeking a QA Engineer to own the quality bar for AI-powered features, including non-deterministic systems and agent-based workflows. You will define what getting it right means, build tooling, and partner with product and engineering to bring rigor to testing LLM-driven features.

You will work with the QA Manager and broader teams to extend automation coverage, improve observability, and ensure safe, reliable AI functionality across the product.

Qualifications

• 6+ years in QA, SDET, or test automation with production automation shipping.
• Hands-on experience testing LLM-based or agentic systems.
• Shift-left, embedded QA model experience.
• Comfort with modern automation stacks: Playwright, Cypress, or similar; TypeScript or Python.
• Deep experience with test frameworks such as vitest, jest, or pytest.
• API-first testing mindset: REST and Postman or equivalent.
• Familiarity with CI/CD pipelines; GitHub Actions a plus.

Responsibilities

• Design and maintain evaluation frameworks for LLM outputs and agentic workflows.
• Build test harnesses to catch hallucinations and tool-calling failures before production.
• Define measurable quality criteria for agent reliability and latency.
• Integrate evaluation runs into CI/CD gates like code changes.
• Collaborate on observability and tracing for agent runs; diagnose failures.
• Contribute to API and end-to-end automation within larger product flows.
• Help shape and coach QA practices as AI features roll out across teams.

Skills

QA Automation

LLM Testing

Shift-Left QA

Playwright

TypeScript

Python

Vitest

Jest

Pytest

REST API testing

CI/CD

Tools

Cypress

Postman

GitHub Actions

Job description

Novara is seeking a QA Engineer to own the quality bar for AI-powered features, including non-deterministic systems and agent-based workflows. You will define what getting it right means, build tooling, and partner with product and engineering to bring rigor to testing LLM-driven features.

You will work with the QA Manager and broader teams to extend automation coverage, improve observability, and ensure safe, reliable AI functionality across the product.