GenAI Tester

Iris Software Inc.; Toronto; Hybrid; CAD 90,000 - 130,000; Full time

Job summary

Iris Software Inc. is seeking a GenAI Tester in Toronto, ON, for a long-term contract. Hybrid work requires 4 days onsite weekly and involvement in AI feature testing within production pipelines.

The ideal candidate has 3+ years in QA, strong Python skills, hands-on GenAI/LLM testing experience, and comfort with API testing, CI/CD, and AI evaluation tools.

Qualifications

• 3+ years in software testing with Python.
• Hands-on experience testing LLM or GenAI applications.
• Strong grasp of LLM concepts: prompts, embeddings, context windows, temperature, RAG.
• Experience with eval tools such as DeepEval, Ragas, Promptfoo, LangSmith, or TruLens.

Responsibilities

• Define the GenAI testing strategy and quality metrics (accuracy, groundedness, hallucination, safety).
• Agentic AI Framework, swarm testing, Agent orchestration – Claude/Devin.
• Build LLM evaluation suites, including LLM-as-a-judge, and plug them into our existing automation framework.
• Create and maintain golden datasets and adversarial prompt sets.
• Test RAG pipelines for retrieval quality, context relevance, and answer faithfulness.
• Validate AI agents and tool-calling flows across multi-step scenarios.
• Track model and prompt regressions across releases and share findings with ML and product teams.
• Upskill the wider QA team on GenAI testing practices.

Skills

Python

GenAI testing

LLM testing

API testing

CI/CD

Tools

DeepEval

Ragas

Promptfoo

LangSmith

TruLens

LangChain

LlamaIndex

Job description

Iris's client, one of the Top 5 Bank in Canada is looking to hire an GenAI Tester for a long-term contract opportunity.

Our client is a Canadian multinational financial services company and the largest bank in Canada by market capitalization. The bank serves over 17 million clients and has more than 89,000 employees worldwide. Bank is serving individual consumers, small and middle market businesses and large corporations with a full range of banking, investing, asset management and other financial and risk-management products and services.

Position: GenAI Tester

Location: Toronto, ON (Hybrid - 4 Day onsite/week)

Duration: Long-Term Open-Ended Contract

Must have:

GenAI test specialist to bring deep expertise in testing LLM-powered features. You'll own how we evaluate AI quality, working within the existing test frameworks and CI/CD pipelines rather than building them from scratch.

What You'll Do

• Define the GenAI testing strategy and quality metrics (accuracy, groundedness, hallucination, safety).


• Agentic AI Framework, swarm testing, Agent orchestration – Claude/Devin.


• Build LLM evaluation suites, including LLM-as-a-judge, and plug them into our existing automation framework.


• Create and maintain golden datasets and adversarial prompt sets.


• Test RAG pipelines for retrieval quality, context relevance, and answer faithfulness.


• Validate AI agents and tool-calling flows across multi-step scenarios.


• Track model and prompt regressions across releases and share findings with ML and product teams.


• Upskill the wider QA team on GenAI testing practices.

What You Bring

• 3+ years in software testing or QA automation, with working Python skills.


• Hands-on experience testing LLM or GenAI applications.


• Strong grasp of LLM concepts: prompts, embeddings, context windows, temperature, RAG.


• Experience with eval tools such as DeepEval, Ragas, Promptfoo, LangSmith, or TruLens.


• Comfort with API testing and working inside an existing CI/CD setup.

Nice to Have

• LangChain / LlamaIndex and vector database experience


• Knowledge of OWASP Top 10 for LLM Applications or AI red-teaming


• ISTQB CT-AI or similar certification

About Iris Software Inc.

With 4,000+ associates and offices in India, U.S.A. and Canada, Iris Software delivers technology services and solutions that help clients complete fast, far-reaching digital transformations and achieve their business goals. A strategic partner to Fortune 500 and other top companies in financial services and many other industries, Iris provides a value-driven approach - a unique blend of highly-skilled specialists, software engineering expertise, cutting-edge technology, and flexible engagement models. High customer satisfaction has translated into long-standing relationships and preferred-partner status with many of our clients, who rely on our 30+ years of technical and domain expertise to future-proof their enterprises. Associates of Iris work on mission-critical applications supported by a workplace culture that has won numerous awards in the last few years, including Certified Great Place to Work in India; Top 25 GPW in IT & IT-BPM; Ambition Box Best Place to Work, #3 in IT/ITES; and Top Workplace NJ-USA.