Senior AI Engineer - Agentic AI Evaluation

Resaro; München; Vor Ort; EUR 90.000 - 140.000; Vollzeit

Zusammenfassung

Resaro is hiring a Senior AI Engineer to own the agentic evaluation engine for MVP and beyond. You will build the evaluation framework that measures tool-using, multi-step agents and ensure reliability across sessions with strong governance alignment.

You’ll operate hands-on with production-grade code, documenting decisions and enabling scalable, auditable evaluations for EU Act, NIST RMF, and ISO 42001 requirements. You’ll collaborate with global teams in a distributed setting.

Qualifikationen

• 5+ years of professional software engineering, shipping production systems in Python.
• Demonstrated depth in agentic AI evaluation, memory, tool-calling, and multi-step control.
• Knowledge of standard benchmarks (GAIA, SWE-bench, WebArena) and their limitations.

Aufgaben

• Own the agentic-evaluation engine and score agents on key metrics.
• Evaluate stateful, multi-turn behavior across sessions with memory.
• Design trustworthy evaluation, isolating runners, auditing ground truth, testing setup.
• Develop the agentic security surface with red-team and adversarial evaluation.
• Bridge lab evaluations to real deployments and measure governance compliance.
• Translate governance requirements into concrete evaluation logic.

Kenntnisse

Python

Agentic AI

Evaluation

Tool-calling

Memory in agents

Benchmarks understanding

Tools

Kubernetes

GPU/CUDA infra

Parquet/Iceberg

DuckDB

Jobbeschreibung

Resaro is an independent, third‑party AI assurance company. We build the software and run the evaluations that let enterprises and public‑sector bodies deploy AI they can actually trust - across computer vision, LLMs, vision‑language models, and, increasingly, agentic and autonomous systems, for clients in government, defence, and regulated commercial sectors.

Our product, the Approved Intelligence Platform (AIP), is where this becomes real software: customers register the AI systems they want tested, run rigorous evaluations, and produce defensible, auditable evidence and reports. Agentic AI is moving from pilots into operational, high‑stakes deployments faster than anyone can assure it - the overwhelming majority of agent pilots still fail before production, on governance, traceability, and reliability gaps rather than raw capability. Closing that gap is the product. That\’s what you’ll build.

The Role

We\’re hiring a Senior AI Engineer to be our go‑to person for agentic AI evaluation - the technical owner of how AIP measures whether tool‑using, multi‑step agents are fit to deploy. You\’ll build the evaluation engine behind our agentic MVP and the roadmap beyond it, and you\’ll be the person the team turns to when an agentic question is hard. As autonomous and embodied systems mature, you\’ll help extend the same rigour into those emerging areas.

This is a hands‑on individual‑contributor role with domain ownership. You report to the Engineering Manager for the evaluation engine and partner closely with Product, who owns the roadmap. You set the technical bar for this vertical; you don\’t line‑manage.

What You’ll Do

• Own the agentic‑evaluation engine. Build the capability that scores agents on the metrics that matter – intent resolution, tool‑selection and parameterisation accuracy, task adherence, planning quality, tool‑call ordering, response completeness against a reference, and when an agent should defer to a human.
• Evaluate stateful, multi‑turn behaviour – not single prompts. Agents carry memory and act over long horizons; your harnesses must expose how behaviour drifts, degrades, or becomes unsafe across a session.
• Make evaluation trustworthy. Design for evaluator integrity: isolate the runner from the agent, assert on semantics not just structure, count crashed/incomplete tasks as failures, audit ground truth, and build tests that probe the test setup itself. A benchmark you can\’t trust is worse than none.
• Build the agentic‑security surface. Partner on red‑team and adversarial evaluation – tool‑chain exploits, RBAC and permission‑inheritance failures, multi‑turn safety erosion – so we catch what a motivated adversary would.
• Close the lab‑to‑production gap. Design evaluations that surface where agents that look good in a demo fail in a real, customer‑specific deployment – the gap that makes independent assurance necessary.
• Translate governance into engineering. Work with our governance analysts to turn EU AI Act, NIST AI RMF, and ISO 42001 requirements into concrete, testable evaluation logic.
• Set the technical bar for the vertical – through spikes, ADRs, and code review – and keep your reasoning legible so the team can build on it.
• Extend into emerging autonomy. As embodied and other autonomous systems become real evaluation targets, help us adapt the engine to them.

What We’re Looking For

• 5+ years of professional software engineering, shipping and operating production systems (not prototypes) in Python.
• Demonstrated depth in agentic AI – you’ve built, orchestrated, or rigorously evaluated LLM agents, and understand tool‑calling, memory, multi‑step control flow, and modern interoperability (MCP for tools, A2A for agent‑to‑agent) at more than a tutorial level.
• A real grasp of how agents are measured – and how measurement fails. You know the standard execution‑based benchmarks (GAIA, τ²‑bench, OSWorld, SWE‑bench Verified, WebArena) and why a high benchmark score routinely overstates production reliability.
• Strong engineering judgement – you can take an ambiguous, research‑adjacent problem, scope it, decompose it, and drive it to a shipped, tested, observable outcome without close supervision.
• A genuine bias to go deep and stay hands‑on, paired with the communication to be the person others come to for this domain – you document decisions and leave work others can pick up.
• Clear written and verbal communication, and comfort being measured against concrete quarterly outcomes, including across a distributed Singapore‑Europe team.

Nice To Have

• Agentic security / red‑team experience – adversarial testing, jailbreak/tool‑chain exploits, or agent identity and on‑behalf‑of authorisation.
• Evaluation, testing, or assurance tooling for any AI domain — LLM/RAG, agentic, or computer vision – including LLM‑as‑judge pipelines and their failure modes.
• Familiarity with AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001) and translating principles into engineering requirements.
• Emerging embodied / autonomous AI exposure – vision‑language‑action models, simulators, or world‑model approaches (a plus, not a requirement).
• Data‑intensive pipelines with columnar/lakehouse formats (Parquet/Iceberg) and DuckDB or similar.
• Container‑based or serverless execution frameworks (Nuclio or comparable) for running evaluations at scale.
• Kubernetes/Helm and GPU/CUDA infrastructure for model inference.
• Experience where evidence, auditability, and reproducibility were first‑class requirements (regulated, safety‑critical, or defence contexts).

We hire to a high, transparent bar.

We look for production‑grade code (correct, tested, observable, maintainable by others); communication and handover (you write clearly and document decisions);

independent operation (you scope and drive ambiguous problems to shipped outcomes);

and a T‑shaped profile – deep in agentic AI evaluation, broad enough to operate across the stack and the ML‑evaluation domain.