Founding AI Quality & Evaluation Scientist

Finom; Barcelona; On-site; EUR 70,000 - 110,000; Full time

Job summary

Finom is seeking a founding IC for its AI Team focused on quality, evaluation, and telemetry of AI agents powering Finom's internal operations and tools across ~20 core processes.

You will extend evaluation suites, enforce pre-launch quality gates in CI/CD, and work with domain experts to label cases and resolve disagreements. You will build datasets from real user and operational traffic and refine metrics to guide operational decisions.

Qualifications

• 5+ years in Data Science / Applied AI with product-level metric ownership.
• Production LLM experience (RAG, multi-step tool use) in last 1–2 years.
• Fluent Python & SQL: build data pipelines, evaluation harnesses and dbt models.

Responsibilities

• Own and extend offline/online evaluation suites for ~20 internal AI agent processes-datasets.
• Establish pre-launch quality gates in CI/CD before production changes hit production
• Label cases with domain experts and resolve annotator disagreements
• Build test datasets from real user and operational traffic
• Harden statistical methodology: drift, bias, non-determinism, true metric shifts
• Translate quality metrics into operational decisions; weekly syncs with process owners

Skills

Statistical analysis

Hypothesis testing

Confidence intervals

Python

SQL

LLM evaluation

Data storytelling

Education

Data Science experience

Tools

dbt

Claude Code

Cursor

Codex

Job description

Finom is seeking a founding IC for its AI Team focused on quality, evaluation, and telemetry of AI agents powering Finom's internal operations and tools across ~20 core processes.

You will extend evaluation suites, enforce pre-launch quality gates in CI/CD, and work with domain experts to label cases and resolve disagreements. You will build datasets from real user and operational traffic and refine metrics to guide operational decisions.