QA Developer, AI Assistant

Poka EN; Montreal; Remote; CAD 70,000 - 95,000; Full time

Benefits offered by this job

Hybrid work policy

100% remote work option

Group insurance

Retirement plan

MacBook Pro + home office allowance

Learning & development budget

Employee referral program

Job summary

Poka EN is hiring a QA specialist to automate QA for our AI-powered assistant on manufacturing floors. You will build evals, regression tests, and tooling to measure answer quality, cost, and speed.

You’ll define acceptance criteria and work with developers to implement fixes, with a focus on automation over manual testing. The role requires 2–4 years in QA/SDET, experience with Python and TypeScript, and a curious mindset about LLM behavior.

Qualifications

• Bachelor-level or equivalent in a related field (AEC/DEC/BAC).
• 2–4 years of relevant QA or SDET experience.
• Experience in test automation and AI/LLM testing.

Responsibilities

• Automate QA for new features and build evaluation tools.
• Create eval suites from real usage data and edge cases.
• Define good answers, write rubrics, and use LLMs as judges where applicable.
• Run CI regression checks on prompts, tools, and model versions.
• Review real conversations and categorize failures for fixes.
• Contribute to adversarial testing and prompt injections.
• Perform exploratory testing on new agents and tools, ensuring safety.

Skills

Python

TypeScript

Test automation

SDET

LLM testing

Education

AEC/DEC or BAC in related field

Tools

Jira

AWS

Docker

Job description

At Poka, we're changing how the world's industrial teams work — capturing the knowledge of their best people and putting it in every operator's hands.

Our connected worker platform is purpose-built for manufacturing and the wider industrial world, with industrial AI inside the shift, not on top of it. The result: frontline performance that becomes compounding knowledge and measurable impact — every shift, every site, every region.

We're proud to support global leaders like Nestlé, Bosch, Mars, and Danone across 3,000+ sites in 70+ countries. And as a fast-growing part of the IFS family, you'll join a team whose daily work directly shapes the platform behind how modern manufacturing gets done.

Role overview

Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.

As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant's answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what "good" means, then you build the evals, automated tests and tools that check it.

How this is different from traditional QA

• No manual test passes. If something needs to be checked more than once, you automate it, with code, AI or whatever tool fits.
• A large part of what you test has no single answer. The same question can get 50 good answers phrased differently, and the assistant can answer differently from one run to the next. You help define what "good" means and measure it.
• You write Python every week.

If you're looking for a role focused on test plans and manual regression testing, this isn't it.

What you'll be responsible for

• Automating QA for the team's features and building the tools that make it easier, including ways for non-developers to add evals and analyze the results.
• Building and maintaining eval suites based on real usage: actual user questions, expected behavior and the edge cases that show up in production.
• Defining with the team what a good answer looks like for our users, writing scoring rubrics and using LLM-as-judge where it makes sense, calibrated against human-labeled samples.
• Running regression checks in CI every time we change a prompt, a tool, retrieval or the Claude model version, covering answer quality, cost and speed. You build the guardrails, evals and tests that tell us whether a release gets the green light or not.
• Reviewing real conversations and grouping failures into patterns: made-up answers, wrong tool calls, missed context, inappropriate tone. Then working with developers to turn them into fixes.
• Contributing to adversarial testing: prompt injection, data leaking between customers, getting the assistant to do things it shouldn't.
• Doing timeboxed exploratory testing on features that change what the assistant can decide or do, like new agents, tools or actions, including checking that actions stay safe when something fails or gets retried.
• Joining feature kickoffs to help define acceptance criteria, including AI-specific ones.

What you bring

• AEC, DEC or BAC in a field related to the position, or equivalent experience
• 2 to 4 years of relevant experience
• A background in test automation or as an SDET
• You're comfortable with several programming languages, including Python and TypeScript
• Real curiosity about how LLMs behave and why they fail
• You're comfortable judging quality when there's no single answer, and explaining your reasoning
• Curiosity about our users' world. You can't judge an answer without understanding what a supervisor on a manufacturing floor actually needs.

Assets

• You've helped set up an eval harness, even in a personal project
• Experience using AI tools to speed up testing
• Basic statistics: pass rates, variance, sample size
• Experience in security or adversarial testing
• Knowledge of the test pyramid and the shift-left approach
• B2B experience or experience on industrial platforms
• Knowledge of Jira

The technologies you'll use

• Python
• TypeScript
• AWS
• Docker
• And many others

We strongly believe that a passion for new technologies, a positive attitude and real team spirit are key to succeeding at Poka.

Above all, we're looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.

If that sounds like you, you're probably the person we're looking for!

What we offer

• Scale-up energy with huge backing and a global yet close-knit team.
• Generous vacation policy, additional paid days, and charity day.
• Hybrid work policy - Employees who live within a 40 km radius of our Québec City office or a 15 km radius of our Montréal office are required to work in-office two days per week.
• 100% remote work if you do not live within the radiuses mentioned above.
• A truly flexible schedule that allows you to adapt your work hours.
• Group insurance from day one (dental, medication, disability, travel, etc.) with employer contributions, including access to telemedicine and an Employee Assistance Program (EAP).
• A retirement savings plan with employer contributions.
• A MacBook Pro and a $500 home office allowance upon hiring.
• A generous annual learning and development budget to support your growth, plus the tools and opportunities you need to keep progressing in your career.
• A rewarding employee referral program for successful hires.
• Tasty snacks available at all times in our offices (fresh fruits, snacks, pastries, coffee, tea, etc.)

*Certain benefits are exclusively accessible to employees in Canada.

Please note: An understanding of Englishis essential at Poka, as it is the language used during company-wide meetings and many internal communications, ensuring effective collaboration and decision-making. In addition, as Poka operates in an international environment, English is the primary language for communicating with our clients and partners, enabling us to better serve them and represent Poka in a professional manner.

At Poka, we are committed to equity, diversity, and inclusion in our hiring. You may choose to share your gender identity below to help us track and improve our hiring practices. This information is entirely optional, will be accessed only by authorized HR personnel, and will be used solely for anonymized, aggregate statistical reporting to ensure equitable hiring. Your choice will remain confidential and will not affect your application in any way. You may withdraw your consent at any time by contacting [e-mail].