Senior AI Evaluation Engineer (CICD & Platform)

Workday, Inc. (old)

Ireland

On-site

EUR 110,000 - 150,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Workday, Inc. is seeking a Senior AI Evaluation Engineer to own the continuous evaluation architecture and release gating for our enterprise AI Skills platform.

You will build MLOps/CI-CD deployment pipelines and automated evaluation harnesses to ensure production AI skills execute safely and remain schema‑compliant across live tenant environments. You will bridge AI skill development with production platform engineering, leveraging modern evaluation frameworks to assess trajectories, test

Qualifications

  • 1+ years of hands-on experience evaluating or building Generative AI applications.
  • Experience with evaluation frameworks and observability tools.
  • 4+ years of professional software engineering in Python—CI/CD OR AI evaluation, plus familiarity with the other.
  • Schema validation libraries (JSON Schema, Pydantic) and test automation.
  • Strong engineering discipline in test coverage, regression prevention, and documentation.

Responsibilities

  • Own the continuous evaluation architecture and release gates for the enterprise AI Skills platform.
  • Build MLOps/CI-CD deployment pipelines and automated eval harnesses.
  • Bridge AI skill development with production platform engineering and governance.
  • Architect and maintain CI/CD build pipelines, versioning, and runner environments for automated gates.
  • Ingest production traces into benchmark datasets to close test coverage gaps.

Skills

Generative AI evaluation
CI/CD pipelines
Python production systems
Technical communication

Education

Bachelor's degree in CS/Engineering

Tools

LangSmith
Braintrust
DeepEval
Promptfoo
Phoenix

Job description

  • The Forward Intelligence team owns the skills that power Workday’s deployment agent—turning the deep configuration expertise and hard-earned wisdom of our Domain Experts and Deployment Consultants into production-grade, fully tested skills that the agent can execute lawfully in live customer tenants
  • What we own: The end-to-end skill engine and governance ecosystem, spanning skill knowledge extraction, promotion pipelines, evaluation frameworks, self-correcting drift detection, and developer authoring environments
  • How we build: Automated testing harnesses and release-gating pipelines running at enterprise scale, ensuring deterministic execution, strict schema compliance, and zero regressions for skills executing in live customer tenants
  • Agentic engineering: We build evaluation tooling, synthetic datasets, and automated red-teaming pipelines that help engineers test how AI agents and Model Context Protocol (MCP) skills perform across complex tenant environments
  • How we work: Close end-to-end collaboration across domain experts, AI platform engineers, and site reliability teams to transform domain knowledge into trustworthy, production-ready AI skills
  • We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform
  • In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments
  • You will bridge the gap between AI skill development and production platform engineering—leveraging modern evaluation frameworks (e.g., LangSmith, Braintrust, DeepEval) alongside CI/CD pipelines to evaluate agent trajectories, test schema contracts, and certify every skill before deployment
  • Continuous Evaluation:. Build tiered evaluation pipelines—separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates
  • CI/CD Quality Gates & MLOps Infrastructure: Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated “pass/fail” release gates on every pull request
  • Deterministic Code & Schema Validation:. Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge‑case evaluation suites from raw API schemas and domain specifications
  • Benchmark Datasets & Regression Testing: Curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks
  • Production Observability: Ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps
  • 1+ years of hands‑on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing)
  • AI Evaluation Tooling: Direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix)
  • AI Security & Governance: Understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries)
  • 4+ years of professional software engineering experience in Python building production‑grade systems— with hands‑on strength in EITHER CI/CD pipeline engineering OR AI evaluation methodology, plus working familiarity with the other
  • Static Code Analysis: Experience using schema validation frameworks (Pydantic, JSON Schema), linters, and test automation tooling to enforce agreed code contracts
  • Technical Communication & Leadership: Demonstrated ability to own problem domains end‑to‑end, collaborate closely with platform and product teams, and write clear technical documentation and architecture designs
  • Agentic Standards & MCP: Familiarity with the Model Context Protocol (MCP), tool‑calling specifications, or agent execution harnesses
  • Schema Contracts & API Validation: Strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing
  • Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience
  • 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates
  • Quality & Testing Discipline: Strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Applied AI Engineer
Senior Applied AI Engineer

Workday, Inc. (old) • Ireland

On-site
EUR 90,000 - 150,000
Applied AI Engineer
Applied AI Engineer

Workday, Inc. (old) • Ireland

On-site
EUR 90,000 - 120,000
Senior AI Evaluation Engineer - CI/CD & Platform
Senior AI Evaluation Engineer - CI/CD & Platform

Workday, Inc. (old) • Ireland

On-site
EUR 110,000 - 150,000
Senior Developer Experience Engineer (AI Platform)
Senior Developer Experience Engineer (AI Platform)

Workday, Inc. (old) • Ireland

On-site
EUR 110,000 - 150,000
Senior Developer Experience (DevEx) Engineer – AI Platform
Senior Developer Experience (DevEx) Engineer – AI Platform

Workday, Inc. (old) • Dublin

On-site
EUR 85,000 - 130,000
AI Native SW Engineering Consultant
AI Native SW Engineering Consultant

Accenture UK & Ireland • Dublin

On-site
EUR 90,000 - 130,000
AI Native SW Engineer - Manager
AI Native SW Engineer - Manager

Accenture UK & Ireland • Dublin

On-site
EUR 90,000 - 130,000
AI Engineer
AI Engineer

beatmysalary • Dublin

On-site
EUR 100,000 - 140,000
Senior AI Evaluation Engineer — CI/CD & Platform
Senior AI Evaluation Engineer — CI/CD & Platform

Workday, Inc. (old) • Dublin

Hybrid
EUR 96,000 - 144,000
Stock grants
Flexible work options
Forward Deployed Solutions Engineer (Applied AI)
Forward Deployed Solutions Engineer (Applied AI)

ServiceNow • Dublin

Hybrid
EUR 120,000 - 180,000
Generous family leave
Matched donations
Annual learning stipends
+3
Top locations