QA Engineer

Ai Makers; France; Ibrido; EUR 60.000 - 90.000; Tempo pieno

Descrizione del lavoro

Ai Makers recherche un QA Engineer pour les systèmes d’IA en production. Vous écrivez et maintenez une suite d’évaluation et un jeu de tests pour un agent support client.

Vous cassez des workflows n8n pour corriger des bugs tout en garantissant que les tests régressent et vous êtes le dernier gate avant la mise en production, validant les chiffres et les indicateurs clé. Vous surveillez les dérives et rédigez des rapports clairs pour une audience non technique.

Competenze

• 3 à 8 ans d’expérience en QA, test engineering ou SDET, avec une partie en systèmes d’IA en production.
• Conception d’évaluations pour des systèmes IA et gestion de non-déterminisme.
• Capacité à produire des rapports d’acceptation clairs et des drift write-ups.
• À l’aise à dire non sous pression et à stopper les livraisons si nécessaire.

Mansioni

• Vous écrivez une suite d’évaluation et un jeu de tests pour un agent support client.
• Vous cassez un workflow n8n pour corriger un bug et vous assurez que les tests régressent.
• Vous êtes le dernier gate avant le live et vous validez les chiffres et les indicateurs de fiabilité et de coût.
• Vous surveillez les dérives et écrivez des rapports compréhensibles par une audience non technique.

Conoscenze

Evaluation techniques

Python

Testing non-deterministic AI outputs

Regression testing

Documentation writing

Stakeholder communication

Strumenti

promptfoo

DeepEval

n8n workflows

CI/CD

Descrizione del lavoro

# QA Engineer : 200 systems in production, and they have to hold upPast a certain number of systems in production, the constraint is no longer building. It is guaranteeing that it holds. That is this role, and it is new. A useful clarification: this is QA for AI systems, not interface testing. You test non-deterministic outputs, not buttons.Postuler/ Le poste## What you take on### You write an evaluation suite, and it liesYou build a test set for a client's support agent. It passes at 95%. Two weeks later a user asks a question phrased unlike anything in your test set, and the system answers nonsense. Your job: maintaining a test set that keeps closing in on real traffic, week after week.### You break an n8n workflow for good reasonsAnd check that nothing else breaks with it. A node is changed to fix a bug; a downstream step that depended on a precise output format fails silently. Your regression tests catch that before the client does.### You are the last gate before deliveryBusiness test set, the client's edge cases, checking that the outputs are presentable to a leader who will use them to decide. You say yes or no to going live.### You verify a number the company repeats everywhere“7 hours a week recovered on average” is a central commercial argument, and a verifiable claim. You build the method to audit it project by project, and you say when it does not hold for a given case.### You watch for what degrades without warningQuality dropping over several weeks with no alert, token costs blowing up. You build the signals that make those drifts visible before the client sees them.### You write a lot, and to be understood immediatelyAcceptance reports, non-conformity notes, drift write-ups: readable in thirty seconds by someone who has not followed the project./ Your first 90 daysMonth 1, you audit what exists and tell us where it breaks most often. Month 2, you put the evaluation foundation in place on a live system. Month 3, no system ships to a client without going through your acceptance./ Le profil## Ce qu'on chercheEvaluation techniques: Golden datasets, scoring rubrics, LLM-as-judge calibrated against human annotation, regression testingTools: Solid Python, an eval framework (promptfoo, DeepEval or equivalent), reading and testing n8n workflows, CI/CDNon-determinism: Thinks in acceptable quality ranges, tells normal variation from a real regressionWriting: Clear reports, readable by a non-technical audienceField work: 3 to 8 years in QA, test engineering or SDET, part of it on AI systems in production, comfortable saying no under pressureCe qu'on ne cherche pasSomeone who executes test plans written by others. Here, you write the strategy./ Le contexte## Where it differs from neighbouring roles| Role | What they do | Who owns the outcome || --- | --- | --- || Generalist QA or manual tester | Predefined scenarios on deterministic software | Functional coverage owns the outcome || MLOps or Platform Engineer | Infrastructure that runs the models | Technical availability owns the outcome || Data Scientist or ML Engineer | Designs and trains the models | Upstream model performance owns the outcome || QA Engineer, AI Systems, AI Makers | Builds the evaluations, decides go or no-go | You own the reliability of the published numbers |/ WhereRemote, in a time zone compatible with France and Morocco (Europe, North Africa or nearby), with no residency requirement, and occasional time in our offices.## Is this role for you?No cover letter. The form takes five minutes and starts with showing what you have built.