Senior AI Evaluation Engineer — CI/CD & Platform

Workday, Inc.; Dublin; Hybrid; EUR 96,000 - 144,000; Full time

Benefits offered by this job

Stock grants

Flexible work options

Job summary

Workday, Inc. in Ireland seeks a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform.

You will build ML deployment pipelines and automated eval harnesses to ensure production skills stay accurate and schema‑compliant. You will bridge AI skill development with production platform engineering, leveraging evaluation frameworks and CI/CD pipelines to certify skills

Qualifications

• 4+ years of professional software engineering in Python building production-grade systems.
• 2+ years designing, building, and operating test automation frameworks (pytest) or CI/CD pipelines.
• 1+ years evaluating or building Generative AI applications (LLM tool-calling, agent workflows).

Responsibilities

• Continuous Evaluation: build tiered evaluation pipelines for PR and release gates.
• CI/CD Quality Gates & MLOps Infrastructure: maintain CI/CD pipelines and runner environments with automated pass/fail gates.
• Deterministic Code & Schema Validation: create synthetic data generation and edge-case evaluation suites from API schemas.
• Benchmark Datasets & Regression Testing: curate versioned golden datasets and run regression tests across model updates.
• Production Observability: feed production traces into benchmarks to close test coverage gaps.

Skills

Python

CI/CD pipelines

AI evaluation

Education

Bachelor's degree in Computer Science, Engineering, or a related field

Tools

GitHub Actions

GitLab CI

Job description

Workday, Inc. in Ireland seeks a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform.

You will build ML deployment pipelines and automated eval harnesses to ensure production skills stay accurate and schema‑compliant. You will bridge AI skill development with production platform engineering, leveraging evaluation frameworks and CI/CD pipelines to certify skills