Applied AI engineer, evaluation

Adverant, Inc.; Dublin; Hybrid; EUR 60.000 - 90.000; Vollzeit

Zusammenfassung

Adverant Limited in Ireland is seeking a role focused on designing and running evaluations for AI agents, ensuring honest and reliable performance. You will create golden sets to keep models honest, and craft quality reports that customers and regulators can verify.

You will own parts of the evaluation workflow, build pipelines to compare model families, and turn problematic results into robust test cases. Collaboration with platform engineers will help keep reviews affordable.

Qualifikationen

• Hands-on experience evaluating language models or ML systems.
• Strong Python and good statistical judgement.
• Experience building tools other engineers rely on.
• Scepticism about single numbers.
• Clear writing.

Aufgaben

• Design and run evaluations for agents, skills and models.
• Build review pipelines that pair models from different families.
• Turn bad results reported by people into lasting test cases.
• Report quality in ways customers and auditors can check.
• Work with platform engineers to make reviews cheap enough to run often.

Kenntnisse

Hands-on experience evaluating LM/ML

Python

Statistical judgement

Tooling

Clear writing

Tools

Python

Jobbeschreibung

Make sure the AI in Nexus does complete, honest work. Build the reviews, test sets and checks that catch problems before customers do.

AI agents on Nexus file claims notes, draft credit notes and classify customs lines. Each one has to be measured, reviewed and kept within its limits, and the measures have to be ones a regulator would accept.

You will own parts of honest review, the golden sets that keep models honest, and the way quality is reported to customers.

What you’ll do

• Design and run evaluations for agents, skills and models.
• Build review pipelines that pair models from different families.
• Turn bad results reported by people into lasting test cases.
• Report quality in ways customers and auditors can check.
• Work with platform engineers to make reviews cheap enough to run often.

What you bring

• Hands‑on experience evaluating language models or ML systems.
• Strong Python and good statistical judgement.
• Experience building tools other engineers rely on.
• Scepticism about single numbers.
• Clear writing.

Good to have

• Work in a regulated field, or on model risk.
• Experience with human review programmes.
• Familiarity with the EU AI Act.

What we offer

Real ownership of a product used in serious places, a team that writes things down, and the time to do the work properly. The written offer sets out pay, level and benefits.

How we hire for this role

• A first call with the hiring manager, about thirty minutes.
• Your work. Walk us through something you built, or take a short exercise.
• Meet the team in a working session on a real problem.
• An offer, in writing.

We give you an answer at every step.