AI Engineer

Fitura; Utrecht; On-site; EUR 90,000 - 120,000; Full time

Job summary

Fitura in the Netherlands is seeking an AI Engineer - RAG Platform Optimization to own model quality across production RAG systems, focusing on retrieval, grounding, and prompt strategies.

You will run experiments in Databricks, analyze telemetry, and build CI/CD data pipelines on Azure to scale improvements for thousands of internal users.

The role combines IR, data science, and software engineering, with a strong emphasis on measurable results and production-grade implementation.

Qualifications

• Master's degree in a quantitative field
• 3+ years of professional experience in ML, AI, or data science
• Strong Python and Databricks experience
• Experience with production-grade RAG systems
• English proficiency in business context

Responsibilities

• Perform failure analysis on RAG outputs and run large-scale experiments.
• Build automated GenAI evaluation pipelines measuring precision/recall and hallucinations.
• Analyze telemetry data in Databricks to detect regressions.
• Translate results into scalable Python production code and CI/CD data pipelines on Azure.
• Collaborate with engineers and product managers to embed metrics into release pipelines.

Skills

Python

Databricks

LLM architecture

Azure cloud environments

GenAI evaluation design

CI/CD pipelines

Education

Master's degree (WO) or equivalent in Data Science, AI, CS, Math, Statistics

Tools

MLflow

TruLens

Ragas

LangSmith

Job description

Role Overview

As an AI Engineer - RAG Platform Optimization within banks central AI capability, you take ownership of continuously improving the accuracy, retrieval quality, and response effectiveness of the bank's enterprise Retrieval-Augmented Generation (RAG) platform. Operating at the convergence of Information Retrieval, Data Science, and Software Engineering, your core mission is failure analysis, hypothesis-driven experimentation, and evaluation framework optimization. Rather than managing people or building standard web UI applications, you focus strictly on measurable model quality-analyzing production telemetry in Databricks, running chunking/embedding experiments, building continuous feedback loops, and ensuring every release raises the performance benchmark for thousands of internal users.

Key Responsibilities:

• Continuous RAG Model Optimization: Perform failure analysis on low-quality responses, identify underlying retrieval or grounding flaws, and run large-scale experiments on chunking, embedding models, prompt optimization, and reranking strategies.

• GenAI Evaluation & Measurement Frameworks: Build and enhance automated evaluation pipelines to measure retrieval precision/recall, response correctness, hallucination rates, and user satisfaction metrics.

• Data Analysis & Telemetry Insights: Analyze user interaction data, production telemetry, and evaluation datasets within Databricks to detect performance regressions and uncover quality improvement opportunities.

• Production Pipeline Implementation: Translate data-driven experimentation results into scalable, maintainable Python production code and automated CI/CD data pipelines on Azure.

• Cross-Functional Technical Alignment: Partner with Software Engineers, Data Scientists, Platform Engineers, and Product Managers to embed evaluation metrics into automated release pipelines.

Your Deliverables:

• Automated GenAI evaluation suites and synthetic benchmark datasets tracking retrieval and response quality.

• Optimized chunking, embedding, reranking, and prompt strategies deployed directly into production pipelines.

• Production telemetry dashboards in Databricks tracking model performance metrics, error categories, and user trust indicators.

• CI/CD-integrated testing scripts that prevent quality regressions prior to production deployments.

Your Profile:

• Education (Hard Requirement): Master's degree (WO) or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics, or a related quantitative field.

• Experience & Track Record (Hard Requirements):

  • Minimum of 3+ years of professional experience in machine learning, applied AI, or data science.

  • Demonstrable track record of improving production-grade RAG systems or advanced retrieval-based AI solutions.

  • Strong hands‑on proficiency in Python, Databricks, LLM architecture, and Azure cloud environments.

  • Proven experience designing evaluation methodologies, quality metrics, and continuous feedback loops for GenAI systems.

  • Proficiency in written and spoken English (corporate business language).

• Nice‑to‑Have Technical Experience:

  • Experience with fine‑tuning language models and reranking algorithms.

  • Hands‑on experience with AI observability tools (e.g., MLflow, TruLens, Ragas, LangSmith).

  • Knowledge of synthetic evaluation dataset creation and benchmark curation.