Senior AI Engineer - Agentic AI Evaluation Brain Team · Munich, Singapore ·

Resaro; München; Vor Ort; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Resaro is hiring a senior engineer to own the agentic evaluation engine for their Approved Intelligence Platform (AIP). You will measure tool-calling, memory, and multi-turn planning, ensuring safe and effective evaluations for complex AI agents.

This hands-on role requires deep domain knowledge and the ability to document decisions for a distributed team in a regulated context. You will work with Product and Governance to translate EU AI Act and NIST RMF requirements into concrete evaluation

Qualifikationen

• 5+ years of professional software engineering in Python and production systems.
• Experience with agentic AI evaluation, LLM agents, tool-calling, and memory.
• Familiar with standard benchmarks (GAIA, SWE-bench, WebArena) and measurement limits.
• Strong ability to scope ambiguities and deliver shipped, tested outcomes.
• Hands-on with domain depth and ability to document decisions for others.

Aufgaben

• Own the agentic-evaluation engine and score agents on key metrics.
• Evaluate stateful, multi-turn behaviour and detect drift or safety issues.
• Design trustworthy evaluation, isolate runner, audit ground truth and tests.
• Build agentic-security surface with red-teaming and permission checks.
• Close lab-to-production gap for customer-specific deployments.
• Translate governance requirements (EU AI Act, NIST RMF, ISO 42001) into tests.

Kenntnisse

Python

Agentic AI evaluation

Benchmark knowledge

Engineering judgement

Hands-on domainExpert

Cross-team communication

Tools

Nuclio

Kubernetes

DuckDB

Parquet/Iceberg

Jobbeschreibung

About Us

Resaro is an independent, third-party AI assurance company. We build the software and run the evaluations that let enterprises and public-sector bodies deploy AI they can actually trust - across computer vision, LLMs, vision-language models, and, increasingly, agentic and autonomous systems, for clients in government, defence, and regulated commercial sectors.

Our product, the Approved Intelligence Platform (AIP), is where this becomes real software: customers register the AI systems they want tested, run rigorous evaluations, and produce defensible, auditable evidence and reports. Agentic AI is moving from pilots into operational, high-stakes deployments faster than anyone can assure it - the overwhelming majority of agent pilots still fail before production, on governance, traceability, and reliability gaps rather than raw capability. Closing that gap is the product. That's what you'll build.

The Role

We're hiring a senior engineer to be our go-to person for agentic AI evaluation - the technical owner of how AIP measures whether tool-using, multi-step agents are fit to deploy. You'll build the evaluation engine behind our agentic MVP and the roadmap beyond it, and you'll be the person the team turns to when an agentic question is hard. As autonomous and embodied systems mature, you'll help extend the same rigour into those emerging areas.

This is a hands-on individual-contributor role with domain ownership. You report to the Engineering Manager for the evaluation engine and partner closely with Product, who owns the roadmap. You set the technical bar for this vertical; you don't line-manage.

What You'll Do

• Own the agentic-evaluation engine. Build the capability that scores agents on the metrics that matter - intent resolution, tool-selection and parameterisation accuracy, task adherence, planning quality, tool-call ordering, response completeness against a reference, and when an agent should defer to a human.




• Evaluate stateful, multi-turn behaviour - not single prompts. Agents carry memory and act over long horizons; your harnesses must expose how behaviour drifts, degrades, or becomes unsafe across a session.




• Make evaluation trustworthy. Design for evaluator integrity: isolate the runner from the agent, assert on semantics not just structure, count crashed/incomplete tasks as failures, audit ground truth, and build tests that probe the test setup itself. A benchmark you can't trust is worse than none.




• Build the agentic-security surface. Partner on red-teaming and adversarial evaluation - tool-chain exploits, RBAC and permission-inheritance failures, multi-turn safety erosion - so we catch what a motivated adversary would.




• Close the lab-to-production gap. Design evaluations that surface where agents that look good in a demo fail in a real, customer-specific deployment - the gap that makes independent assurance necessary.




• Translate governance into engineering. Work with our governance analysts to turn EU AI Act, NIST AI RMF, and ISO 42001 requirements into concrete, testable evaluation logic.




• Set the technical bar for the vertical - through spikes, ADRs, and code review - and keep your reasoning legible so the team can build on it.




• Extend into emerging autonomy. As embodied and other autonomous systems become real evaluation targets, help us adapt the engine to them.

What We're Looking For

• 5+ years of professional software engineering, shipping and operating production systems (not prototypes) in Python.




• Demonstrated depth in agentic AI - you've built, orchestrated, or rigorously evaluated LLM agents, and understand tool-calling, memory, multi-step control flow, and modern interoperability (MCP for tools, A2A for agent-to-agent) at more than a tutorial level.




• A real grasp of how agents are measured - and how measurement fails. You know the standard execution-based benchmarks (GAIA, τ²-bench, OSWorld, SWE-bench Verified, WebArena) and why a high benchmark score routinely overstates production reliability.




• Strong engineering judgement - you can take an ambiguous, research-adjacent problem, scope it, decompose it, and drive it to a shipped, tested, observable outcome without close supervision.




• A genuine bias to go deep and stay hands-on, paired with the communication to be the person others come to for this domain - you document decisions and leave work others can pick up.




• Clear written and verbal communication, and comfort being measured against concrete quarterly outcomes, including across a distributed Singapore-Europe team.

Nice To Have

• Agentic security / red-teaming experience - adversarial testing, jailbreak/tool-chain exploits, or agent identity and on-behalf-of authorisation.




• Evaluation, testing, or assurance tooling for any AI domain — LLM/RAG, agentic, or computer vision - including LLM-as-judge pipelines and their failure modes.




• Familiarity with AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001) and translating principles into engineering requirements.




• Emerging embodied / autonomous AI exposure - vision-language-action models, simulators, or world-model approaches (a plus, not a requirement).




• Data-intensive pipelines with columnar/lakehouse formats (Parquet/Iceberg) and DuckDB or similar.




• Container-based or serverless execution frameworks (Nuclio or comparable) for running evaluations at scale.




• Kubernetes/Helm and GPU/CUDA infrastructure for model inference.




• Experience where evidence, auditability, and reproducibility were first-class requirements (regulated, safety-critical, or defence contexts).

Our Hiring Blueprint

We hire to a high, transparent bar. We look for production-grade code (correct, tested, observable, maintainable by others); communication and handover (you write clearly and document decisions); independent operation (you scope and drive ambiguous problems to shipped outcomes); and a T-shaped profile - deep in agentic AI evaluation, broad enough to operate across the stack and the ML-evaluation domain.

Resaro is an Equal Opportunity Employer. We respect each individual and support the diverse cultures, perspectives, skills and experiences within our teams.