Agentic AI Quality Engineer

3900 Accenture B.V. Company; Amsterdam; Hybrid; EUR 90,000 - 140,000; Full time

Benefits offered by this job

4x9 Workweek

Hybrid Working

Annual Bonus

Health Insurance

Learning & Development Platform

Mobility Budget

Vacation Days 26

Do Good Volunteer Program

Job summary

Accenture NL seeks an Agentic AI Quality Engineer to lead evaluation and safety testing for next-generation conversational AI. You will join multidisciplinary teams of quality engineers, software engineers, AI specialists, and business consultants to advance reliable, compliant AI solutions.

The role emphasizes defining measurable quality criteria, building reusable assets, and guiding go/no-go decisions with stakeholders across Digital Core and related service lines.

Qualifications

• Education: Bachelor's or Master's in Computer Science, Data Science, or related field.
• Experience: 3+ years in machine learning QA, AI evaluation engineering, or MLOps engineering.
• Languages: Dutch – B2+; English – C2.
• Skills: Agentic AI expertise: evaluating autonomous agents, multi-step workflows, or RAG applications.
• Tech: Strong Python; experience with ML evaluation tools such as MLflow, Galileo, PyRIT.
• Frameworks: Knowledge of prompt engineering, vector databases, tool schema design, and telemetry.
• Security: Hands-on adversarial testing, threat modeling, or model safety guardrails.
• Leadership: Strong communication and growth mindset for innovation.

Responsibilities

• Automate evaluation of multi-turn conversations and long-horizon tasks.
• Implement safety guardrails and human-in-the-loop reviews for high-risk cases.
• Run red-team, prompt-injection, jailbreak, and privacy tests across workflows.
• Embed test harnesses in CI/CD and monitor production telemetry for drift.
• Communicate findings and recommendations to stakeholders and coaches teams.

Skills

Agentic AI

Python

AI evaluation

Adversarial testing

Leadership

Education

Bachelor's/Master's in CS/DS

Tools

MLflow

Galileo

PyRIT

Job description

Lead with expertise, drive innovation. Are you an experienced professional ready to take your career to the next level? At Accenture the Netherlands, we're seeking an Agentic AI Quality Engineer who brings deep expertise, strategic thinking, and a passion for innovation to help us solve complex challenges and deliver exceptional results.

What you’ll do:

• Quality & Performance Evaluation Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, evidence grounding, plan execution, latency, token usage, and recovery. Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse.
• Safety & Compliance Guardrails Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift. Use human-in-the-loop review for high-risk edge cases.
• Adversarial Security & Red Teaming Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows. Analyze failures in model logic and API integrations to identify root causes and controls.
• MLOps & Production Monitoring Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery. Maintain evaluation data lineage, versioning, curation, and golden responses.
• Consulting & Collaboration Translate business risks into measurable quality criteria, practical controls, and release recommendations. Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy. Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards.

What Success Looks Like (KPIs)

• Safety: Zero critical production incidents.
• Automation: High coverage of non-deterministic agent paths.
• Speed: Rapid regression detection and remediation after model updates.

Industry Context & Methodology Alignment

Treat non-determinism as a core characteristic. As the Data Science Collective notes, robust testing verifies correct reasoning, grounded evidence, scoped actions, and safe, compliant decisions.

Drive continuous optimization by embedding evaluation in CI/CD, using tunable judges, aligned scorers, automated dataset updates, and dashboards, as outlined by Databricks.

How We Reinvent

At Accenture, our teams operate within one of our core service lines— Digital Core, Industry & Enterprise, Supply Chain & Engineering, Cybersecurity, Song, Finance, Talent, Client Groups, Global Operations, Reinventions Services —each designed to drive reinvention at scale. This role falls under Digital Core, where we help clients modernize their technology foundations and accelerate business transformation through cloud, data, software engineering, and AI.

You will join multidisciplinary teams consisting of quality engineers, software engineers, architects, AI specialists, platform engineers, and business consultants. Together, we help organizations reinvent how software is designed, built, tested, deployed, and operated through the power of AI. We go to market by industry, leveraging our global delivery network and strategic partnerships with ecosystem leaders like Microsoft, AWS, and Google.

Our culture of shared success and continuous learning ensures that every team member contributes to—and benefits from—our reinvention journey.

What we’re looking for:

• Education: Bachelor's or Master's in Computer Science, Data Science, or a related quantitative field.
• Experience: 3+ years in machine learning QA, AI evaluation engineering, or MLOps engineering.
• Language(s): Dutch – B2+ for collaboration with Dutch-speaking clients and stakeholders. English – C2.
• Skills: Agentic AI Expertise: Experience evaluating autonomous agents, multi-step workflows, or Retrieval-Augmented Generation (RAG) applications.
• Technical Stack: Strong Python proficiency and experience with AI evaluation tools such as MLflow, Galileo, and PyRIT.
• Framework Mastery: Deep knowledge of prompt engineering, vector databases, tool schema design, and execution telemetry.
• Security Mindset: Hands-on experience in adversarial testing, threat modeling, or model safety guardrails.
• Leadership: Strong communication, technical leadership, a growth mindset, and commitment to continuous innovation.

Bonus points for:

• TMAP or ISTQB Certification
• Contributions to open-source projects or engineering communities

What we offer:

• Work-Life Flexibility: 4x9 Workweek Option, Hybrid Working, FlexDay
• Financial Well-being: Competitive Base Salary, MyBenefits Budget, Annual Bonus, Net Allowance, Employee Share Purchase Plan
• Health & Wellness: Collective Health Insurance Scheme, Mental Health Support, Well-being Hub
• Time Off & Leave: 26 Vacation Days, Special Leave Types (parental leave, partner leave, bereavement leave, and gender affirmation leave)
• Culture Days
• Mobility & Commuting: NS Business Card, Mobility Budget (option to choose between electric bike, car lease, or public transport reimbursement)
• Career Development: Performance Achievement Program, Learning & Development Platform, Guidance from People Leads
• Inclusion, Diversity & Social Impact: Do Good Volunteer Program, I&D Networks, Refugee Talent Track
• Culture & Community: Accentrics & Friday Drinks, Annual Accenture NL Party, Flat Organizational Hierarchy

Equal Employment Opportunity Statement

We believe that no one should be discriminated against because of their differences.All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, sexual orientation, gender identity or expression, marital status, citizenship status or any other basis as protected by applicable law.Our rich diversity makes us more innovative, more competitive, and more creative, which helps us better serve our clients and our communities.

About Accenture

Accenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services—creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. We combine our strength in technology and leadership in cloud, data and AI with unmatched industry experience, functional expertise, and global delivery capability. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360° value, enable us to help our clients reinvent and build trusted, lasting relationships. We measure our success by the 360° value we create for our clients, each other, our shareholders, partners and communities. Visit us at www.accenture.com

Explore your next step at Accenture Belong. Grow. Thrive. Join agreat place to work for reinventors who drive meaningful change for our clients, communities, and the world.