Agentic AI QA Engineer: Safety, Automation & MLOps

Accenture the Netherlands; Amsterdam; On-site; EUR 70,000 - 100,000; Full time

Benefits offered by this job

4x9 Workweek Option

Hybrid Working

Annual Bonus

Health Insurance

Net Allowance

Employee Share Purchase Plan

Job summary

Accenture the Netherlands is seeking an Agentic AI Quality Engineer to drive quality across AI evaluation, safety guardrails, and adversarial testing. You will automate multi-turn conversation evaluation, implement tools for detection of tool-call issues, and lead red-team exercises to ensure robust, compliant AI deployments.

You will collaborate with cross-functional teams to embed evaluation in CI/CD, monitor production telemetry, and maintain data lineage and golden responses.

Qualifications

• Education: Bachelor's or Master's in Computer Science, Data Science, or a related quantitative field.
• Experience: 3+ years in machine learning QA, AI evaluation engineering, or MLOps engineering.
• Language(s): Dutch – B2+ for collaboration with Dutch-speaking clients and stakeholders; English – C2.

Responsibilities

• Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, evidence grounding, plan execution, latency, token usage, and recovery.
• Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse.
• Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift.
• Use human-in-the-loop review for high-risk edge cases.
• Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows.
• Analyze failures in model logic and API integrations to identify root causes and controls.
• Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery.
• Maintain evaluation data lineage, versioning, curation, and golden responses.
• Translate business risks into measurable quality criteria, practical controls, and release recommendations.
• Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy.
• Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards.
• Safety: Zero critical production incidents.
• Automation: High coverage of non-deterministic agent paths.
• Speed: Rapid regression detection and remediation after model updates.

Skills

Agentic AI Expertise

Leadership

Security Mindset

Framework Mastery

Education

Bachelor's or Master's in Computer Science/Data Science/related field

Tools

MLflow

Galileo

PyRIT

Job description

Accenture the Netherlands is seeking an Agentic AI Quality Engineer to drive quality across AI evaluation, safety guardrails, and adversarial testing. You will automate multi-turn conversation evaluation, implement tools for detection of tool-call issues, and lead red-team exercises to ensure robust, compliant AI deployments.

You will collaborate with cross-functional teams to embed evaluation in CI/CD, monitor production telemetry, and maintain data lineage and golden responses.