QA Developer, AI Assistant

Poka EU; Montreal; Hybrid; CAD 65,000 - 90,000; Full time

Benefits offered by this job

Hybrid work policy

Remote options

Group insurance

MacBook Pro + home office allowance

Learning and development budget

Employee referral program

Job summary

Poka is hiring a QA Automation Engineer to drive automated testing for our AI assistant used on manufacturing floors. You will design evals, build tools, and work with developers to ensure robust, safe behavior across prompts and tools.

You’ll automate QA, define scoring rubrics, and run regression checks in CI. You’ll collaborate with product, design, and engineering teams to ship high‑quality AI capabilities.

Qualifications

• 2–4 years of relevant experience in QA automation or SDET.
• Experience with test automation tools and frameworks and scripting languages.
• Curiosity about LLM behavior and failure modes.
• Ability to define acceptance criteria and scoring rubrics for AI-driven features.
• Comfortable explaining reasoning when there isn’t a single correct answer.

Responsibilities

• Automate QA for features and build eval tools for non-developers to add evals.
• Build and maintain eval suites based on real usage and edge cases.
• Define what a good answer looks like and calibrate with human samples.
• Run CI regression checks for prompts, tools, and model versions.
• Review real conversations and group failures to drive fixes with the team.
• Contribute to adversarial testing and prompt injection tests.
• Perform timeboxed exploratory testing for feature changes.
• Join feature kickoffs to define acceptance criteria, including AI-specific ones.

Skills

Python

TypeScript

QA Automation

SDET

Education

AEC/DEC/BAC in related field

Tools

Claude Code

promptfoo

AWS

Docker

Jira

Job description

At Poka, we’re changing how the world’s industrial teams work — capturing the knowledge of their best people and putting it in every operator’s hands.

Our connected worker platform is purpose-built for manufacturing and the wider industrial world, with industrial AI inside the shift, not on top of it. The result: frontline performance that becomes compounding knowledge and measurable impact — every shift, every site, every region.

We’re proud to support global leaders like Nestlé, Bosch, Mars, and Danone across 3,000+ sites in 70+ countries. And as a fast-growing part of the IFS family, you’ll join a team whose daily work directly shapes the platform behind how modern manufacturing gets done.

Role overview

Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.

As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant’s answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what “good” means, then you build the evals, automated tests and tools that check it.

How this is different from traditional QA

• No manual test passes. If something needs to be checked more than once, you automate it, with code, AI or whatever tool fits.
• A large part of what you test has no single answer. The same question can get 50 good answers phrased differently, and the assistant can answer differently from one run to the next. You help define what “good” means and measure it.
• You write Python every week.

If you’re looking for a role focused on test plans and manual regression testing, this isn’t it.

What you’ll be responsible for

• Automating QA for the team’s features and building the tools that make it easier, including ways for non-developers to add evals and analyze the results.
• Building and maintaining eval suites based on real usage: actual user questions, expected behavior and the edge cases that show up in production.
• Defining with the team what a good answer looks like for our users, writing scoring rubrics and using LLM-as-judge where it makes sense, calibrated against human-labeled samples.
• Running regression checks in CI every time we change a prompt, a tool, retrieval or the Claude model version, covering answer quality, cost and speed. You build the guardrails, evals and tests that tell us whether a release gets the green light or not.
• Reviewing real conversations and grouping failures into patterns: made-up answers, wrong tool calls, missed context, inappropriate tone. Then working with developers to turn them into fixes.
• Contributing to adversarial testing: prompt injection, data leaking between customers, getting the assistant to do things it shouldn’t.
• Doing timeboxed exploratory testing on features that change what the assistant can decide or do, like new agents, tools or actions, including checking that actions stay safe when something fails or gets retried.
• Joining feature kickoffs to help define acceptance criteria, including AI-specific ones.

What you bring

• AEC, DEC or BAC in a field related to the position, or equivalent experience
• 2 to 4 years of relevant experience
• A background in test automation or as an SDET
• You’re comfortable with several programming languages, including Python and TypeScript
• Real curiosity about how LLMs behave and why they fail
• You’re comfortable judging quality when there’s no single answer, and explaining your reasoning
• Curiosity about our users’ world. You can’t judge an answer without understanding what a supervisor on a manufacturing floor actually needs.

Assets

• You’ve helped set up an eval harness, even in a personal project
• Experience using AI tools to speed up testing
• Basic statistics: pass rates, variance, sample size
• Experience in security or adversarial testing
• Knowledge of the test pyramid and the shift-left approach
• B2B experience or experience on industrial platforms
• Knowledge of Jira

The technologies you’ll use

• Python
• TypeScript
• Claude Code
• Eval framework (promptfoo)
• AWS
• Docker
• And many others

We strongly believe that a passion for new technologies, a positive attitude and real team spirit are key to succeeding at Poka.

Above all, we’re looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.

If that sounds like you, you’re probably the person we’re looking for!

What we offer:

• Scale-up energy with huge backing and a global yet close-knit team.
• Generous vacation policy, additional paid days, and charity day.
• Hybrid work policy - Employees who live within a 40 km radius of our Québec City office or a 15 km radius of our Montréal office are required to work in‑office two days per week.
• 100% remote work if you do not live within the radiuses mentioned above.
• A truly flexible schedule that allows you to adapt your work hours.
• Group insurance from day one (dental, medication, disability, travel, etc.) with employer contributions, including access to telemedicine and an Employee Assistance Program (EAP).
• A retirement savings plan with employer contributions.
• A MacBook Pro and a $500 home office allowance upon hiring.
• A generous annual learning and development budget to support your growth, plus the tools and opportunities you need to keep progressing in your career.
• A rewarding employee referral program for successful hires.
• Tasty snacks available at all times in our offices (fresh fruits, snacks, pastries, coffee, tea, etc.)

*Certain benefits are exclusively accessible to employees in Canada.

Please note: An understanding of Englishis essential at Poka, as it is the language used during company-wide meetings and many internal communications, ensuring effective collaboration and decision‑making. In addition, as Poka operates in an international environment, English is the primary language for communicating with our clients and partners, enabling us to better serve them and represent Poka in a professional manner.