Senior Software Quality Engineer – AI

PeoplePilot; Toronto; On-site; CAD 95,000 - 130,000; Full time

Job summary

PeoplePilot seeks a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role collaborates with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks in a Toronto onsite setting.

The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery.

Qualifications

• 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation.
• 5+ years of programming experience with Java, Python, Groovy, or similar languages.
• Strong experience building and maintaining test automation frameworks.
• Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured.
• Experience with API, database, integration, regression, and microservices testing.
• Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter.
• Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications.
• Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency.
• Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing.
• Strong analytical, problem-solving, communication, and documentation skills.

Responsibilities

• Develop and execute comprehensive quality strategies, test plans, test cases, and test data.
• Design, develop, and maintain scalable test automation frameworks.
• Create automated tests for AI/ML and Generative AI features.
• Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency.
• Test APIs, microservices, databases, integrations, data pipelines, and third-party services.
• Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
• Integrate automated testing and AI quality checks into CI/CD pipelines.
• Identify, document, prioritize, and track defects through resolution.
• Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
• Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.

Skills

QA

SDET

Test Automation

Java

Python

Groovy

Selenium

Playwright

Cucumber

RestAssured

JUnit

TestNG

QA Frameworks

API Testing

CI/CD

Data Testing

Tools

Selenium

Playwright

Cucumber/Gherkin

RestAssured

JUnit

TestNG

JMeter

ReadyAPI

qTest

Job description

We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON.

Key Requirements

• 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation.
• 5+ years of programming experience with Java, Python, Groovy, or similar languages.
• Strong experience building and maintaining test automation frameworks.
• Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured.
• Experience with API, database, integration, regression, and microservices testing.
• Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter.
• Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications.
• Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency.
• Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing.
• Strong analytical, problem-solving, communication, and documentation skills.

Key Responsibilities

• Develop and execute comprehensive quality strategies, test plans, test cases, and test data.
• Design, develop, and maintain scalable test automation frameworks.
• Create automated tests for AI/ML and Generative AI features.
• Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency.
• Test APIs, microservices, databases, integrations, data pipelines, and third-party services.
• Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
• Integrate automated testing and AI quality checks into CI/CD pipelines.
• Identify, document, prioritize, and track defects through resolution.
• Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
• Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.