GenAI QA Tester - Python, LLM Eval, Hybrid Toronto

Iris Software Inc.; Toronto; Hybrid; CAD 70,000 - 110,000; Full time

Job summary

Iris Software Inc. is seeking an AI Tester in Toronto, on a hybrid schedule (4 days in office, 1 day remote) for a long-term contract with a banking client. The role focuses on QA automation, Python-based tests, and hands-on GenAI/LLM testing with experience in eval tools and API validation.

The successful candidate will work within Iris’s engineering teams to ensure robust AI features pass through the CI/CD pipeline and meet banking-grade quality standards.

Qualifications

• 3+ years in software testing or QA automation
• Working Python skills required
• Hands-on experience testing LLM or GenAI applications
• Strong grasp of LLM concepts: prompts, embeddings, context windows, temperature, RAG
• Experience with eval tools such as DeepEval, Ragas, Promptfoo, LangSmith, or TruLens
• Comfort with API testing and working inside an existing CI/CD setup

Responsibilities

• Lead testing activities for AI/GenAI features in the banking client environment.
• Collaborate with devs to develop test strategies and plans.
• Implement and maintain test suites across APIs and integrations.
• Validate LLM prompts, embeddings, and context handling.
• Participate in CI/CD pipeline validation and regression testing.

Skills

Python

LLM/GenAI testing

API testing

CI/CD

QA automation

Tools

DeepEval

Ragas

Promptfoo

LangSmith

TruLens

Job description

Iris Software Inc. is seeking an AI Tester in Toronto, on a hybrid schedule (4 days in office, 1 day remote) for a long-term contract with a banking client. The role focuses on QA automation, Python-based tests, and hands-on GenAI/LLM testing with experience in eval tools and API validation.

The successful candidate will work within Iris’s engineering teams to ensure robust AI features pass through the CI/CD pipeline and meet banking-grade quality standards.