Senior ML Engineer - Evaluations Platform (Remote)

Prosus

Amsterdam

Hybrid

EUR 120,000 - 180,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Relocation support
Amsterdam office base
Remote work arrangements
Competitive compensation
MacBook Pro

Job summary

Prosus is seeking a Senior Machine Learning Engineer to own the development of evaluations and the platform that runs them. You will define meaningful tasks, curate evaluation data, implement reliable scoring, investigate model behavior and provide actionable insights for research and product decisions.

In a flat team of around 30 engineers, you will work with applied scientists, training engineers and domain experts to build, automate and scale evaluation capabilities across model development

Qualifications

  • At least five years of ML or software engineering experience, including substantial hands-on work building and evaluating ML systems.
  • Experience owning evaluation frameworks, benchmarks or model-quality systems that informed training or deployment decisions.
  • Strong Python and experience with PyTorch, Transformers or comparable ML libraries, alongside reliable production software development.
  • Practical knowledge of experimental design and statistics, including sampling, uncertainty, repeated trials and comparisons across model versions.
  • Experience running inference or evaluation workloads at scale and debugging failures across models, data, scoring code and infrastructure.
  • Clear communication and the ability to turn ambiguous results into defensible recommendations.

Responsibilities

  • Design evaluations with scientists and domain experts, translating intended capabilities into representative tasks, datasets and success criteria. Cover current applications and capabilities we want future models to acquire.
  • Curate evaluation data and calibrate task difficulty, so that evaluations are neither saturated nor out of reach and still separate models of different quality. Work with domain experts on sourcing, annotation and review, within data provenance, permitted use across companies and privacy requirements.
  • Own the development and operation of the evaluation platform across training checkpoints, deployed models and external baselines. Support distributed execution, reproducible configurations, trace inspection and comparison of results, while managing runtime and cost.
  • Develop programmatic checks, outcome-based scoring and LLM-based graders. Calibrate automated judgments against expert reviews, investigate disagreements and test whether graders reward superficial answers or reject valid solutions.
  • Analyse model outputs and agent trajectories, including tool calls, environment state and task outcomes. Distinguish model failures from problems in the evaluation, quantify uncertainty and explain what changed between versions.
  • Maintain versioned tasks, datasets, graders and environments, with protected held-out tests and contamination checks. Expand coverage as models improve, retaining useful regression tests and adding harder tasks where existing tests no longer distinguish performance.
  • Prepare commerce benchmarks for public release on ProLLM, including public mirror sets and evaluation code, where the data can be shared.
  • Integrate evaluations into training and release workflows, with dashboards and regression checks that support concrete decisions. Work with portfolio teams to establish whether offline improvements correspond to better outcomes in online experiments.
  • Build tools and agents that help generate candidate tasks, run evaluations and investigate failures, validating their outputs before adoption. Review contributions and mentor engineers so teams can extend the suite consistently.

Skills

Python
PyTorch
Transformers
Production software development
Experimental design
Communication

Job description

Prosus is seeking a Senior Machine Learning Engineer to own the development of evaluations and the platform that runs them. You will define meaningful tasks, curate evaluation data, implement reliable scoring, investigate model behavior and provide actionable insights for research and product decisions.

In a flat team of around 30 engineers, you will work with applied scientists, training engineers and domain experts to build, automate and scale evaluation capabilities across model development

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Machine Learning Engineer, Evals
Senior Machine Learning Engineer, Evals

Prosus • Amsterdam

Hybrid
EUR 120,000 - 180,000
Relocation support
Amsterdam office base
Remote work arrangements
+2
Senior ML Engineer - Remote (NL) & Growth
Senior ML Engineer - Remote (NL) & Growth

Jobgether • Netherlands

Remote
EUR 90,000 - 150,000
Remote work options
Medical insurance
Learning budget
+1
Senior Data Engineer - AI Training Data Platform
Senior Data Engineer - AI Training Data Platform

Prosus • Amsterdam

Hybrid
EUR 90,000 - 130,000
Relocation support
Remote work arrangements
MacBook Pro
+1
Senior MLOps Engineer - Flexible Hours, Scalable AI
Senior MLOps Engineer - Flexible Hours, Scalable AI

Relx Plc • Amsterdam

On-site
GBP 70,000 - 110,000
Pension Plan
Sabbatical leave
Parental leave (Maternity/Paternity/Ad
+2
Senior ML Engineer, Applied ML & Research
Senior ML Engineer, Applied ML & Research

super • Netherlands

On-site
EUR 120,000 - 170,000
Medical / Health Insurance
Open Annual Leave
Employee Assistance Programme
+1
Senior ML Engineer: Build Scalable Production Models
Senior ML Engineer: Build Scalable Production Models

Adyen • Amsterdam

Hybrid
EUR 90,000 - 140,000
Remote LLM Evaluation Engineer - AI Research & Benchmarks
Remote LLM Evaluation Engineer - AI Research & Benchmarks

Blue Lynx Employment BV • Netherlands

Remote
EUR 60,000 - 80,000
25 holidays per annum
Pension plan
Opportunity to work on cutting-edge AI initiatives
Senior AI Engineer - Production ML & Guardrails (Remote)
Senior AI Engineer - Production ML & Guardrails (Remote)

Engaige • Utrecht

Hybrid
EUR 90,000 - 130,000
Stock options
Growth opportunities toward CTO
Senior ML Engineer: Scalable, Production-Ready Recommendations
Senior ML Engineer: Scalable, Production-Ready Recommendations

Y-Axis • Zuid-Holland

Hybrid
EUR 90,000 - 135,000
Senior ML Engineer - Agentic Search, Remote
Senior ML Engineer - Agentic Search, Remote

pofft Technology Inc. • Amsterdam

Remote
EUR 140,000 - 200,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+3