AI / ML QA Engineer

Infotek Consulting Inc.; Toronto; On-site; CAD 85,000 - 110,000; Full time

Job summary

A consulting firm is seeking an experienced AI/ML QA Engineer in Toronto to support model testing and validation for advanced machine learning systems. The role involves designing test cases, validating model outputs, and conducting adversarial testing for bias and compliance. Ideal candidates will have hands-on experience with AI model testing, proficiency in Python, and knowledge of test automation frameworks. Familiarity with LLMs and prompt engineering is a plus. Contract duration is from May 4 to November 4, 2026.

Qualifications

• Hands on experience testing AI and ML models including NLP, classification, and generative AI.
• Strong experience with LLM testing including prompt engineering and evaluation of hallucinations, bias, and response quality.
• Proficiency in Python and test automation frameworks.

Responsibilities

• Design and execute test cases for AI and ML models including NLP, classification, recommendation, and generative AI systems.
• Validate outputs for accuracy, consistency, and expected behavior across diverse datasets and inputs.
• Conduct adversarial testing to uncover vulnerabilities such as prompt injection and jailbreak scenarios.

Skills

AI/ML model testing

Prompt engineering

Python

Test automation frameworks

CI/CD pipelines

Regression testing

Tools

Python-based testing frameworks

Job description

Contract Duration: May 4, 2026 to November 4, 2026

Overview

We are seeking an experienced AI / ML QA Engineer to support testing and validation of advanced machine learning and generative AI systems. This role focuses on ensuring model quality, reliability, and responsible AI practices across a range of applications including NLP, classification, recommendation systems, and large language models.

Key Responsibilities

Functional and Model Testing

• Design and execute test cases for AI and ML models including NLP, classification, recommendation, and generative AI systems
• Validate outputs for accuracy, consistency, and expected behavior across diverse datasets and inputs
• Perform regression testing following model updates, retraining cycles, or data changes

Prompt and LLM Testing

• Develop and maintain prompt test libraries for LLM based features
• Evaluate responses for relevance, tone, factual accuracy, and alignment with business rules
• Identify hallucinations, response drift, and edge case failures

AI Safety and Red Teaming

• Conduct adversarial testing to uncover vulnerabilities such as prompt injection and jailbreak scenarios
• Assess outputs for bias, fairness, and compliance with responsible AI standards
• Document risks and collaborate with model teams on remediation strategies

Test Automation and Tooling

• Build and maintain automated test suites using Python based frameworks
• Track and report quality metrics, model performance benchmarks, and defect trends

Collaboration and Documentation

• Work closely with data scientists, ML engineers, and product teams
• Develop clear test plans, defect reports, and evaluation summaries
• Contribute to QA standards and best practices for AI systems

Required Skills

• Hands on experience testing AI and ML models including NLP, classification, and generative AI
• Strong experience with LLM testing including prompt engineering and evaluation of hallucinations, bias, and response quality
• Proficiency in Python and test automation frameworks
• Experience working with CI CD pipelines
• Solid understanding of regression testing

Nice to Have

• Experience in banking or financial services
• Exposure to AI safety and red teaming techniques such as prompt injection and jailbreak testing
• Knowledge of responsible AI and model governance principles
• Familiarity with LLM evaluation tools and frameworks
• Experience testing recommendation or ranking systems
• Exposure to ML pipelines and MLOps workflows
• Understanding of bias and fairness metrics

Note: We use AI tools to: obtain basic information, detect plagiarism, false employment history or references, categorize your skills, and do an initial match with job posting.