Agentic AI Quality Engineer: Safe & Scalable Testing

Accenture; Amsterdam; On-site; EUR 90,000 - 140,000; Full time

Job summary

Accenture Netherlands seeks an Agentic AI Quality Engineer to lead the evaluation and governance of AI systems, focusing on safety, accuracy, and operational excellence. You will design tests, drive automated evaluation, and collaborate with multidisciplinary teams to translate business risks into measurable quality criteria.

You will embed evaluation in CI/CD, implement golden datasets, and oversee red-team testing, ensuring robust, trusted AI in client projects across the Digital Core domain.

Qualifications

• Experience evaluating AI systems and performance metrics.
• Understanding of safety, compliance, and bias considerations in AI.
• Familiarity with CI/CD, testing, and observability concepts.

Responsibilities

• Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, evidence grounding, plan execution, latency, token usage, and recovery.
• Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse.
• Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift.
• Use human-in-the-loop review for high-risk edge cases.
• Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows.
• Analyze failures in model logic and API integrations to identify root causes and controls.
• Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery.
• Maintain evaluation data lineage, versioning, curation, and golden responses.
• Translate business risks into measurable quality criteria, practical controls, and release recommendations.
• Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy.
• Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards.
• Drive continuous optimization by embedding evaluation in CI/CD, using tunable judges, aligned scorers, automated dataset updates, and dashboards.

Job description

Accenture Netherlands seeks an Agentic AI Quality Engineer to lead the evaluation and governance of AI systems, focusing on safety, accuracy, and operational excellence. You will design tests, drive automated evaluation, and collaborate with multidisciplinary teams to translate business risks into measurable quality criteria.

You will embed evaluation in CI/CD, implement golden datasets, and oversee red-team testing, ensuring robust, trusted AI in client projects across the Digital Core domain.