Senior AI Engineer - Agentic AI Evaluation

Resaro AI; München; Hybrid; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Resaro AI is hiring a senior engineer to own the agentic evaluation engine for the Approved Intelligence Platform (AIP). You will score agentic AI systems on key metrics, handle multi-turn stateful behavior, and ensure rigorous, auditable evaluation.

You will work with product and governance to translate EU AI Act requirements into concrete tests. The role is hands-on, requires deep domain knowledge in agentic AI, and involves collaboration across a distributed Singapore-Europe team and modern

Qualifikationen

• 5+ years shipping production software in Python.
• Deep experience with agentic AI and LLM agents.
• Understanding of tool-calling, memory, multi-step control flow.
• Demonstrated ability to scope ambiguous problems to shipped outcomes.
• Strong written and verbal communication across distributed teams.

Aufgaben

• Own the agentic-evaluation engine and score agents on key metrics.
• Evaluate stateful, multi-turn behaviour over long horizons.
• Design for evaluator integrity and auditable tests.
• Develop red-teaming and adversarial evaluation capabilities.
• Close the lab-to-production gap for customer deployments.
• Translate governance standards into tests (EU AI Act, NIST RMF, ISO 42001).
• Set technical bar through spikes, ADRs, and code reviews.
• Extend engine to embodied/autonomous systems.

Kenntnisse

Python

Agentic AI

Tool-calling

LLM agents

Engineering judgement

Communication

Tools

Parquet

Iceberg

DuckDB

Kubernetes

GPU/CUDA

Jobbeschreibung

About Us

Resaro is an independent, third-party AI assurance company. We build the software and run the evaluations that let enterprises and public-sector bodies deploy AI they can actually trust - across computer vision, LLMs, vision-language models, and, increasingly, agentic and autonomous systems, for clients in government, defence, and regulated commercial sectors.

Our product, the Approved Intelligence Platform (AIP), is where this becomes real software: customers register the AI systems they want tested, run rigorous evaluations, and produce defensible, auditable evidence and reports. Agentic AI is moving from pilots into operational, high-stakes deployments faster than anyone can assure it - the overwhelming majority of agent pilots still fail before production, on governance, traceability, and reliability gaps rather than raw capability. Closing that gap is the product. That's what you'll build.

The Role

We're hiring a senior engineer to be our go-to person for agentic AI evaluation - the technical owner of how AIP measures whether tool-using, multi-step agents are fit to deploy. You'll build the evaluation engine behind our agentic MVP and the roadmap beyond it, and you'll be the person the team turns to when an agentic question is hard. As autonomous and embodied systems mature, you'll help extend the same rigour into those emerging areas.

This is a hands-on individual-contributor role with domain ownership. You report to the Engineering Manager for the evaluation engine and partner closely with Product, who owns the roadmap. You set the technical bar for this vertical; you don't line-manage.

What You'll Do

• Own the agentic-evaluation engine. Build the capability that scores agents on the metrics that matter - intent resolution, tool-selection and parameterisation accuracy, task adherence, planning quality, tool-call ordering, response completeness against a reference, and when an agent should defer to a human.
• Evaluate stateful, multi-turn behaviour - not single prompts. Agents carry memory and act over long horizons; your harnesses must expose how behaviour drifts, degrades, or becomes unsafe across a session.
• Make evaluation trustworthy. Design for evaluator integrity: isolate the runner from the agent, assert on semantics not just structure, count crashed/incomplete tasks as failures, audit ground truth, and build tests that probe the test setup itself. A benchmark you can't trust is worse than none.
• Build the agentic-security surface. Partner on red-teaming and adversarial evaluation - tool-chain exploits, RBAC and permission-inheritance failures, multi-turn safety erosion - so we catch what a motivated adversary would.
• Close the lab-to-production gap. Design evaluations that surface where agents that look good in a demo fail in a real, customer-specific deployment - the gap that makes independent assurance necessary.
• Translate governance into engineering. Work with our governance analysts to turn EU AI Act, NIST AI RMF, and ISO 42001 requirements into concrete, testable evaluation logic.
• Set the technical bar for the vertical - through spikes, ADRs, and code review - and keep your reasoning legible so the team can build on it.
• Extend into emerging autonomy. As embodied and other autonomous systems become real evaluation targets, help us adapt the engine to them.

What We're Looking For

• 5+ years of professional software engineering, shipping and operating production systems (not prototypes) in Python.
• Demonstrated depth in agentic AI - you've built, orchestrated, or rigorously evaluated LLM agents, and understand tool-calling, memory, multi-step control flow, and modern interoperability (MCP for tools, A2A for agent-to-agent) at more than a tutorial level.
• A real grasp of how agents are measured - and how measurement fails. You know the standard execution-based benchmarks (GAIA, τ²-bench, OSWorld, SWE-bench Verified, WebArena) and why a high benchmark score routinely overstates production reliability.
• Strong engineering judgement - you can take an ambiguous, research-adjacent problem, scope it, decompose it, and drive it to a shipped, tested, observable outcome without close supervision.
• A genuine bias to go deep and stay hands-on, paired with the communication to be the person others come to for this domain - you document decisions and leave work others can pick up.
• Clear written and verbal communication, and comfort being measured against concrete quarterly outcomes, including across a distributed Singapore-Europe team.

Nice To Have

• Agentic security / red-teaming experience - adversarial testing, jailbreak/tool-chain exploits, or agent identity and on-behalf-of authorisation.
• Evaluation, testing, or assurance tooling for any AI domain — LLM/RAG, agentic, or computer vision - including LLM-as-judge pipelines and their failure modes.
• Familiarity with AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001) and translating principles into engineering requirements.
• Emerging embodied / autonomous AI exposure - vision-language-action models, simulators, or world-model approaches (a plus, not a requirement).
• Data-intensive pipelines with columnar/lakehouse formats (Parquet/Iceberg) and DuckDB or similar.
• Container-based or serverless execution frameworks (Nuclio or comparable) for running evaluations at scale.
• Kubernetes/Helm and GPU/CUDA infrastructure for model inference.
• Experience where evidence, auditability, and reproducibility were first-class requirements (regulated, safety-critical, or defence contexts).

Our Hiring Blueprint

We hire to a high, transparent bar. We look for production-grade code (correct, tested, observable, maintainable by others); communication and handover (you write clearly and document decisions); independent operation (you scope and drive ambiguous problems to shipped outcomes); and a T-shaped profile - deep in agentic AI evaluation, broad enough to operate across the stack and the ML-evaluation domain.