Applied AI Engineer — Evaluation & Quality for Regulated AI

Adverant Limited; Dublin; Hybrid; EUR 60,000 - 90,000; Full time

Job summary

Adverant Limited in Ireland is seeking a role focused on designing and running evaluations for AI agents, ensuring honest and reliable performance. You will create golden sets to keep models honest, and craft quality reports that customers and regulators can verify.

You will own parts of the evaluation workflow, build pipelines to compare model families, and turn problematic results into robust test cases. Collaboration with platform engineers will help keep reviews affordable.

Qualifications

• Hands-on experience evaluating language models or ML systems.
• Strong Python and good statistical judgement.
• Experience building tools other engineers rely on.
• Scepticism about single numbers.
• Clear writing.

Responsibilities

• Design and run evaluations for agents, skills and models.
• Build review pipelines that pair models from different families.
• Turn bad results reported by people into lasting test cases.
• Report quality in ways customers and auditors can check.
• Work with platform engineers to make reviews cheap enough to run often.

Skills

Hands-on experience evaluating LM/ML

Python

Statistical judgement

Tooling

Clear writing

Tools

Python

Job description

Adverant Limited in Ireland is seeking a role focused on designing and running evaluations for AI agents, ensuring honest and reliable performance. You will create golden sets to keep models honest, and craft quality reports that customers and regulators can verify.

You will own parts of the evaluation workflow, build pipelines to compare model families, and turn problematic results into robust test cases. Collaboration with platform engineers will help keep reviews affordable.