AI QA Lead: ML Testing & Automation Expert

Pacer Group; Toronto; Hybrid; CAD 69,000 - 110,000; Full time

Job summary

Pacer Group in Toronto employs a senior-quality engineering role with hybrid work. The team seeks 10+ years in QA, including AI/ML evaluation, and strong Python/test automation/API/integration testing capabilities.

You will design tests for prompts and model outputs, perform regression and non-functional testing, and provide auditable evidence leading to go/no-go decisions for production readiness.

Qualifications

• 10+ years of Quality Engineering experience.
• 2+ years evaluating AI/ML systems.
• Strong Python, test automation, API testing and integration testing.
• Experience designing prompt/response test sets and evaluating accuracy, groundedness, relevance, and hallucinations.
• Hands-on testing: regression, bias/safety, performance, penetration, non-functional.

Skills

Quality Engineering

AI/ML eval

Python

Test automation

API testing

Integration testing

Prompt testing

Regression testing

Bias testing

Performance testing

Non-functional testing

GitHub Actions

GitHub Copilot

Claude Code

Copilot Studio

LangSmith

AppInsights

Log Analytics

Dynatrace

Auditable evidence

Defect analytics

Go/no-go

Job description

Pacer Group in Toronto employs a senior-quality engineering role with hybrid work. The team seeks 10+ years in QA, including AI/ML evaluation, and strong Python/test automation/API/integration testing capabilities.

You will design tests for prompts and model outputs, perform regression and non-functional testing, and provide auditable evidence leading to go/no-go decisions for production readiness.