Agentic AI Quality Engineer

parsionate; Amsterdam; Presencial; EUR 90.000 - 120.000; Jornada completa

Descripción de la vacante

Accenture Netherlands in Amsterdam seeks an Agentic AI Quality Engineer to lead quality and performance evaluation for complex AI systems. You will drive testing, validation, and governance for safe, scalable AI deployments within Digital Core.

You will work with quality engineers, AI specialists, platform engineers, and business consultants to define metrics, implement automated tests, and ensure rapid, safe iterations aligned with client needs and industry standards.

Responsabilidades

• Automate evaluation of multi-turn conversations and long-horizon tasks.
• Apply schema validation, tracing, and calibrated judges to detect issues in tool calls and loops.
• Perform red-team, jailbreak, and privacy tests across agent workflows.
• Monitor production telemetry and maintain data lineage for evaluations.
• Translate business risks into measurable quality criteria and release guidance.
• Communicate findings and residual risks for go/no-go decisions.

Conocimientos

Quality Assurance

MLOps

CI/CD

Threat Modeling

Descripción del empleo

Lead with expertise, drive innovation.

Are you an experienced professional ready to take your career to the next level? At Accenture the Netherlands, we're seeking an Agentic AI Quality Engineer who brings deep expertise, strategic thinking, and a passion for innovation to help us solve complex challenges and deliver exceptional results.

• Quality Engineering Associate Manager | Mid-Level | Full time
• Amsterdam

Job Description

What you'll do:

Quality & Performance Evaluation

• Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, evidence grounding, plan execution, latency, token usage, and recovery.
• Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse.

Safety & Compliance Guardrails

• Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift.
• Use human-in-the-loop review for high-risk edge cases.

Adversarial Security & Red Teaming

• Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows.
• Analyze failures in model logic and API integrations to identify root causes and controls.

MLOps & Production Monitoring

• Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery.
• Maintain evaluation data lineage, versioning, curation, and golden responses.

Consulting & Collaboration

• Translate business risks into measurable quality criteria, practical controls, and release recommendations.
• Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy.
• Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards.

What Success Looks Like (KPIs)

• Safety: Zero critical production incidents.
• Automation: High coverage of non-deterministic agent paths.
• Speed: Rapid regression detection and remediation after model updates.

Industry Context & Methodology Alignment

• Treat non-determinism as a core characteristic. As the Data Science Collective notes, robust testing verifies correct reasoning, grounded evidence, scoped actions, and safe, compliant decisions.
• Drive continuous optimization by embedding evaluation in CI/CD, using tunable judges, aligned scorers, automated dataset updates, and dashboards, as outlined by Databricks.

How We Reinvent

At Accenture, our teams operate within one of our core service lines— Digital Core, Industry & Enterprise, Supply Chain & Engineering, Cybersecurity, Song, Finance, Talent, Client Groups, Global Operations, Reinventions Services —each designed to drive reinvention at scale. This role falls under Digital Core, where we help clients modernize their technology foundations and accelerate business transformation through cloud, data, software engineering, and AI.

You will join multidisciplinary teams consisting of quality engineers, software engineers, architects, AI specialists, platform engineers, and business consultants. Together, we help organizations reinvent how software is designed, built, tested, deployed, and operated through the power of AI.

We go to market by industry, leveraging our global delivery network and strategic partnerships with ecosystem leaders like Microsoft, AWS, and Google. Our culture of shared success and continuous learning ensures that every team member contributes to—and benefits from—our reinvention journey.