Senior AI Evaluation & Quality Scientist (EU Remote)

Finom; Barcelona; Hybride; EUR 70 000 - 110 000; Plein temps

Avantages offerts par ce poste

Stock options

Work & Swim program

Résumé du poste

Finom’s AI Team seeks a seasoned data scientist to own evaluation methodologies for its internal AI agents. You will design benchmarks, set quality gates, and partner with domain experts to label cases and resolve disagreements.

You’ll work with Databricks, Claude Code, and dbt to build robust evaluation pipelines. You will lead the offline and online evaluation suites across ~20 processes and translate findings into actionable product decisions, balancing quality, cost, and latency.

Qualifications

• 5+ years in Data Science / Product Analytics / Applied AI roles with product-level metric ownership.
• Production LLM Experience: built, shipped, or evaluated LLM-based systems as core responsibility in last 1–2 years.
• Autonomous Quality Ownership: owned evaluation methodology or analytics for an entire product or end-to-end process.
• Fluent Python & SQL: write clean data pipelines, evaluation harnesses, and dbt transformations.
• Statistical Rigor: sampling, hypothesis testing, variance analysis, and confidence intervals on noisy metrics.

Responsabilités

• Own and extend offline/online evaluation suites across ~20 internal AI agent processes.
• Establish pre-launch quality gates: enforce pass/fail thresholds in CI/CD pipelines.
• Label cases with domain experts and resolve annotator disagreements.
• Build test datasets from real user and operational traffic.
• Harden statistical methodology and measure metric shifts vs. noise.
• Translate quality metrics into operational decisions with process owners.

Connaissances

Data Science

Product Analytics

Applied AI

Python

SQL

Statistical Rigor

Outils

Databricks

DeepEval

Claude Code

Cursor

Python

SQL

dbt

Description du poste

Finom’s AI Team seeks a seasoned data scientist to own evaluation methodologies for its internal AI agents. You will design benchmarks, set quality gates, and partner with domain experts to label cases and resolve disagreements.

You’ll work with Databricks, Claude Code, and dbt to build robust evaluation pipelines. You will lead the offline and online evaluation suites across ~20 processes and translate findings into actionable product decisions, balancing quality, cost, and latency.