AI Engineer -RAG Platform

1UP Training (Netherlands); Utrecht; Hybrid; EUR 70,000 - 100,000; Full time

Job summary

1UP Training in Utrecht is seeking an AI Engineer – RAG Platform to enhance a production Retrieval-Augmented Generation system. You will work at the intersection of information retrieval, data science and software engineering to improve performance and user satisfaction.

The role focuses on experimentation, evaluation, and scalable deployment. Collaboration with software engineers, data scientists and product managers is essential, with a hybrid work setup in the Netherlands.

Qualifications

• Master's degree or equivalent in a related field.
• At least 3 years of relevant experience in ML, AI or data science.
• Proven experience improving production-grade RAG systems or advanced retrieval-based AI solutions.
• Experience designing evaluation methodologies and quality metrics.
• Ability to communicate findings to technical and non-technical stakeholders.
• Experience with large datasets and distributed environments.

Responsibilities

• Improve retrieval performance, answer quality and user experience through analysis, experimentation and optimization.
• Investigate low-quality answers and identify retrieval or grounding issues.
• Develop automated evaluation frameworks to measure retrieval quality and user satisfaction.
• Run large-scale experiments on chunking, embedding and retrieval strategies.
• Analyze user interactions and system performance data to identify opportunities for improvement.
• Build feedback loops that identify weak responses and inform ongoing development.
• Define and monitor quality metrics to measure improvements and identify regressions.
• Turn analysis and experimentation into scalable, production-ready solutions.

Skills

RAG systems

Python

Databricks

LLMs

Evaluation frameworks

CI/CD pipelines

Education

Master’s degree in Data Science / AI / CS

Tools

Databricks

Azure

Python

Job description

AI Engineer – RAG Platform

Location: Utrecht
Working hours: 36 hours per week
Language: English, written and spoken
Working arrangement: Hybrid

About the role
As an AI Engineer – RAG Platform, you will improve the quality, accuracy and effectiveness of a production Retrieval-Augmented Generation (RAG) platform. You will work at the intersection of information retrieval, data science and software engineering, focusing on the performance of the RAG system itself.
You will investigate where and why the system underperforms, design experiments to test improvements, and translate your findings into measurable production outcomes. You will contribute throughout the improvement lifecycle, from failure analysis and hypothesis generation to evaluation, implementation and monitoring. Your work will draw on user feedback, retrieval metrics, evaluation datasets and production telemetry.
You will collaborate with software engineers, platform engineers, data scientists and product managers. This is a hands-on specialist role focused on retrieval systems, evaluation methodologies and applied AI.

What you’ll do

Improve retrieval performance, answer quality and the overall user experience through analysis, experimentation and optimization.
Investigate low-quality answers and identify retrieval or grounding issues.
Develop automated evaluation frameworks to measure retrieval quality, answer quality and user satisfaction.
Run large-scale experiments on chunking, embedding and retrieval strategies.
Analyze user interactions and system performance data in Databricks to identify opportunities for improvement.
Build feedback loops that identify weak responses and inform ongoing development.
Define and monitor quality metrics to measure improvements and identify regressions.
Turn analysis and experimentation into scalable, maintainable production solutions.

What you bring

A master’s degree or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics or a related field.
At least 3 years of relevant experience in machine learning, applied AI or data science.
Demonstrable experience improving production-grade RAG systems or advanced retrieval-based AI solutions.
Experience designing evaluation methodologies and quality metrics.
A strong track record of turning analysis into automated production solutions.
Experience working with large datasets and complex distributed environments.
The ability to communicate findings and recommendations to technical and non-technical stakeholders.
A curious, analytical approach and a focus on measurable outcomes.

Technical skills

Retrieval-Augmented Generation (RAG) and large language models (LLMs)
Python and Databricks
Evaluation frameworks for generative AI systems
Embedding models and prompt optimization
Data pipelines and workflow automation
CI/CD practices and Azure

Nice to have

Experience fine-tuning language models.
Experience with ranking and retrieval systems.
Experience with AI observability and monitoring.
Experience working with large-scale production AI platforms.
Knowledge of synthetic evaluation datasets and benchmark creation.

Eisen

• English is mandatory
• The candidate must be available throughout the entire duration of the assignment
• All submitted candidates must have a valid passport or ID card and bring it to the interview and the start of the assignment
• The candidate may start only after successfully completing the pre-employment screening
• Midlance constructions are not allowed
• The candidate must have RAG experience
• The candidate must have experience with Large Language Models (LLMs)
• The candidate must have Python skills
• The candidate must have Databricks skills
• The candidate must have experience with evaluation frameworks for Gen
• AI systems
• The candidate must have experience with embedding models
• The candidate must have experience with prompt optimization
• The candidate must have experience with data pipelines and workflow automation
• The candidate must have experience with CI/CD practices
• The candidate must have experience with Azure
• The candidate must have a master's degree or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics, or a related field
• The candidate must have at least 3 years of relevant experience in machine learning, applied AI, or data science
• The candidate must have demonstrable experience improving production-grade RAG systems or advanced retrieval-based AI solutions
• The candidate must have experience designing evaluation methodologies and quality metrics
• The candidate must have a strong track record of converting analyses into automated production solutions
• The candidate must have experience working with large datasets and complex distributed environments
• The candidate must be able to communicate findings and recommendations to both technical and non-technical stakeholders
• The candidate must be proficient in written and spoken English

Gunningscriteria

• Experience with fine-tuning language models is a nice-to-have
• Experience with ranking and retrieval systems is a nice-to-have
• Experience with AI observability and monitoring is a nice-to-have
• Experience working with large-scale production AI platforms is a nice-to-have
• Knowledge of synthetic evaluation datasets and benchmark creation is a nice-to-have