AI QA Evaluation Engineer

Aptiv; Dublin; On-site; EUR 70,000 - 110,000; Full time

Benefits offered by this job

Healthcare

Pension

Free onsite meals

Gym membership

Training & certifications

Job summary

Aptiv in Dublin seeks an AI QA / Evaluation Engineer to define, implement, and continuously improve testing frameworks for enterprise AI agents and workflows. You will collaborate with business stakeholders and engineering teams to ensure solutions are functional, reliable, and compliant before release.

Responsibilities include designing test plans, data sets, and evaluation criteria; coordinating with DevOps, AI Architects, and security/compliance teams; and maintaining test environments to

Qualifications

• Bachelor’s degree in CS or related field.
• Experience in software testing, QA engineering or validation in enterprise environments.
• Strong test design, execution and defect management.
• Experience in agile/Kanban delivery.
• Ability to define structured test approaches for complex systems.
• Knowledge of AI/ML or automation testing is desirable.

Responsibilities

• Define and implement testing and evaluation frameworks for AI agents and agentic workflows, covering functional quality, groundedness, safety, reliability, and performance.
• Develop and execute test plans, test cases, evaluation criteria, test data, datasets, and benchmark scenarios to ensure consistent and repeatable validation.
• Partner with DevOps Agent Engineers and Release / Operations Engineers to validate AI solutions throughout build, integration, pre-production, and deployment phases.
• Work with AI Architects to ensure evaluation approaches align with solution design, risk profiles, enterprise standards, and responsible AI requirements.
• Own and maintain development and test environments for AI agents, ensuring stability, availability, and alignment with enterprise platform standards.
• Support business users during user acceptance testing and validate end-user developed agents prior to publishing, ensuring solutions are fit for purpose and meet enterprise quality and governance requirements.
• Identify, log, and track defects, working closely with engineering teams to support resolution, retesting, and continuous improvement of testing practices, tooling, and automation.
• Collaborate with Cyber Security, Legal, and Data Privacy teams to ensure testing addresses compliance, security, and responsible AI requirements, while maintaining clear documentation of test results, quality metrics, and evaluation outcomes.

Skills

QA testing

Test design

Defect management

Agile/Kanban

Structured testing

AI/ML testing

Problem solving

Education

Bachelor’s degree in Computer Science or related field

Tools

CI/CD pipelines

Scripting

Test automation tools

Copilot Studio

Azure AI Foundry

Job description

AI QA / Evaluation Engineer (Enterprise AI Agents) The QA / Evaluation Engineer will work directly with business stakeholders and engineering teams to define, execute and continuously improve testing and evaluation practices for enterprise AI agents and agentic workflows. Reporting to the AI Platform Manager, this role is responsible for ensuring all AI solutions are functional, reliable, compliant and fit for purpose prior to release. The role owns evaluation frameworks and test standards, and is accountable for maintaining development and test environments to support consistent, high-quality delivery. The role will also liaise directly with business users to support user acceptance testing and to validate end-user built agents ahead of publishing, ensuring alignment with enterprise standards, governance requirements and quality thresholds.

Responsibilities

• Define and implement testing and evaluation frameworks for AI agents and agentic workflows, covering functional quality, groundedness, safety, reliability, and performance.
• Develop and execute test plans, test cases, evaluation criteria, test data, datasets, and benchmark scenarios to ensure consistent and repeatable validation.
• Partner with DevOps Agent Engineers and Release / Operations Engineers to validate AI solutions throughout build, integration, pre-production, and deployment phases.
• Work with AI Architects to ensure evaluation approaches align with solution design, risk profiles, enterprise standards, and responsible AI requirements.
• Own and maintain development and test environments for AI agents, ensuring stability, availability, and alignment with enterprise platform standards.
• Support business users during user acceptance testing and validate end-user developed agents prior to publishing, ensuring solutions are fit for purpose and meet enterprise quality and governance requirements.
• Identify, log, and track defects, working closely with engineering teams to support resolution, retesting, and continuous improvement of testing practices, tooling, and automation.
• Collaborate with Cyber Security, Legal, and Data Privacy teams to ensure testing addresses compliance, security, and responsible AI requirements, while maintaining clear documentation of test results, quality metrics, and evaluation outcomes.

Required Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems or a related field.
• Experience in software testing, QA engineering or validation roles within enterprise technology environments.
• Strong understanding of test design, test execution and defect management practices.
• Experience working with development teams in agile or Kanban-based delivery environments.
• Ability to define structured test approaches for complex systems, including integration and end-to-end scenarios.
• Experience working across engineering, architecture and control functions in a governed enterprise environment.
• Strong analytical and problem-solving skills with attention to detail.
• Understanding of AI/ML or automation solutions and their unique testing considerations is highly desirable.

Preferred Qualifications

• Experience with AI evaluation frameworks, including groundedness, safety, hallucination detection and performance testing.
• Familiarity with Microsoft Copilot Studio, Azure AI Foundry or similar enterprise AI platforms.
• Experience with test automation tools, scripting or CI/CD-integrated testing approaches.
• Exposure to data validation, prompt testing or evaluation dataset design.
• Experience working with enterprise governance, responsible AI, security and compliance requirements.
• Experience operating in global, matrixed or regulated enterprise environments.

Your Benefits at Aptiv: Benefits/Perks

• Personal holidays, Healthcare, Pension, Tax saver scheme, Free Onsite Breakfast & Lunch, Discounted Corporate Gym Membership.
• Multicultural environment Learning, professional growth and development in a world-recognized international environment.
• Access to internal & external training, coaching & certifications.
• Recognition for innovation and excellence.
• Access to transportation: Grand Canal Dock is well-connected to public transportation, including DART trains, buses, and bike-sharing services, making it easy to get to and from the area.

At Aptiv, we deliver innovative solutions that people trust when it matters most. Progress is never static. It is about reaching further, exploring new possibilities, and enabling mobility. Aptiv empowers customers to create solutions people rely on—advancing safety and autonomy in Automotive and Commercial Vehicles, enabling mission-critical performance in Aerospace & Defense, and powering intelligent systems across Energy, Enterprise, and Telecommunications. We also drive precision and efficiency in Industrials & Robotics and support life‑critical innovation in Medical technologies. Guided by our winning culture and values to Think & Act Like Owners, as One Team, and Play to Win, we turn commitment into results and achieve more together.

Aptiv is an equal employment opportunity employer.

All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, gender identity, sexual orientation, disability status, protected veteran status or any other characteristic protected by law.

Aptiv is a global industrial technology company enabling automated, electrified and digitalized solutions across multiple end markets. People trust us to deliver innovations that will help them progress, improve and achieve more tomorrow than they did yesterday.

Please review Aptiv’s privacy policy by following this link: https://www.aptiv.com/privacy-notice