AI Quality Analyst

DutchTech; España; On-site; PHP 4,291,845 - 6,131,208; Full time

Benefits offered by this job

Stock options

Work & Swim program in Cyprus

Flexible work model across Europe

Job summary

DutchTech in the Philippines seeks a specialist in AI safety who will manage the quality and performance of AI systems. Responsibilities include designing automated evaluation frameworks, curating benchmarks, and ensuring high-quality outputs across multi-agent workflows.

The ideal candidate should possess Python coding skills, analytical thinking for testing AI systems, and have a collaborative mindset. Additional perks include remote work flexibility and opportunities for stock options.

Qualifications

• Strong Python coding and software engineering fundamentals.
• Deep understanding of LLMs and analytical testing mindset.
• Experience coordinating across technical teams.

Responsibilities

• Design and maintain scalable evaluation frameworks.
• Collaborate to build and sanitize gold-standard datasets.
• Monitor and enforce quality KPIs for agent workflows.
• Lead analyses to mitigate systemic risks in outputs.
• Integrate automated testing gates with engineering teams.
• Analyze efficiency of models for optimization.

Skills

Python coding experience

Software engineering fundamentals

Analytical thinking

Understanding of token behaviours

Tools

LangSmith

DeepEval

Ragas

Opik

Job description

About Finom

Finom is a European tech startup headquartered in Amsterdam. We’re on a journey towards revolutionizing the financial landscape for entrepreneurs worldwide. Our mission is to develop an all‑in‑one financial B2B solution that integrates banking functions, accounting, financial management, and invoicing into a seamless, mobile‑first platform.

We recently closed a €115 million Series C equity round (around $133 million), bringing our total funding to approximately $346 million. This significant investment follows a $105 million growth funding round from General Catalyst, a long‑term backer since 2021 known for supporting companies like Airbnb, HubSpot, KAYAK, and Stripe.

Finom’s platform goes beyond traditional banking, offering invoicing and a growing suite of features, including AI‑enabled accounting, aiming to simplify financial management for entrepreneurs. We’re actively expanding our reach across key EU markets like Germany, France, the Netherlands, Italy, and Spain.

At Finom, we’re not just redefining the entrepreneurial experience — we’re empowering our employees to make a real difference. Your work matters, and your impact extends far beyond product metrics. We nurture innovation and an inspiring work environment where bold ideas thrive, prioritizing thorough research, swift implementation of solutions, and ensuring that every effort we make benefits our users, employees, partners, and our business as a whole.

In this highly cross‑functional role, you will be the gatekeeper of AI safety, performance, and deterministic behavior across non‑deterministic multi‑agent systems. You will not be filling out generic manual testing spreadsheets or operating in a vacuum.

What You Will Be Doing

• Architect Automated Evaluation Frameworks: Design, implement, and maintain scalable evaluation pipelines (Evals) for LLMs and agent graphs using modern tooling like LangSmith, DeepEval, Ragas, or Opik.
• Curate Ground‑Truth Benchmarks: Collaborate with domain experts to build, version, and sanitize robust gold‑standard datasets, synthetic evaluation profiles, and edge‑case testing matrices reflecting real‑world business scenarios.
• Own Non‑Deterministic Quality Tracking: Define, monitor, and enforce quality KPIs across multi‑agent workflows—specifically focusing on tool‑calling accuracy, intent‑recognition safety, structured output formatting, and context‑retrieval (RAG) precision.
• Mitigate and Quantify Systemic Risk: Lead rigorous failure and hallucination analyses on production outputs. Implement structured LLM‑as‑Judge patterns, validation metrics, and guardrail heuristics while actively ensuring the judge profiles remain free of baseline evaluation bias.
• Enforce CI/CD Evaluation Gates: Partner directly with MLOps and Backend Engineering teams to integrate automated testing gates into our deployment pipelines, proactively preventing regressions or behavioural drifts from reaching production runtime environments.
• Drive Optimization for Latency & Cost: Regularly analyze the efficiency of prompt templates, few‑shot structures, and model selections (e.g., GPT, Claude, LLaMA) to ensure a highly calibrated balance between execution throughput, sub‑second latency, and platform compute costs.

Who You Are

• A Data‑Savvy Automation Advocate: You possess strong software engineering fundamentals and concrete Python coding experience, allowing you to seamlessly script custom evaluation routines and query multi‑tenant databases.
• An Analytical Thinker with an AI Lens: You understand that testing non‑deterministic LLMs requires a completely different mindset than traditional QA. You possess deep intuition for token behaviours, retrieval dynamics, prompt engineering nuances, and failure states.
• Radically Autonomous & Collaborative: You do not wait around for static technical specifications. You independently coordinate syncs with AI leads, domain backend engineers, and product stakeholders to identify and patch system vulnerabilities.
• Rigorously Quality‑Oriented: You hold a low ego but maintain high standards for system stability. You are deeply passionate about separating market hype from practical, measurable production metrics.

What You Will Get In Return

• Make a genuine impact on the product.
• Work in the EU with flexibility to travel, work remotely, or in a hybrid model across Europe.
• Become a stock options holder.
• Receive unwavering support and care from our modern, friendly, and eco‑conscious corporate culture.
• Work & Swim program: spend one month in a comfortable corporate apartment in Cyprus while enjoying Mediterranean views.

Equal Opportunity Statement

At Finom, we’re an equal opportunity employer and value diversity at our company. We embrace diversity and invite applications from all walks of life. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, age, marital status, disability status, or other applicable legally protected characteristics.