Data Scientist: LLM QA Evaluation & Metrics

Kaizo Nederland; Amsterdam; Hybrid; EUR 55,000 - 90,000; Full time

Benefits offered by this job

Visa sponsorship available

Great office gear

Flexible working schedule

Open holiday policy

Fun workations

Team events

Job summary

Kaizo Nederland is seeking a sharp data scientist to own evaluation and quality loops for the AutoQA product. You can be early in your career, with room to grow into Senior Data Scientist or ML Engineer roles.

The role sits at the intersection of AI and CX, collaborating with enterprise customers and teams in Amsterdam. We offer flexible hours, visa sponsorship for eligible candidates, and a strong growth path in a fast-growing SaaS company.

Qualifications

• Strong grounding in evaluating AI/ML models and quality metrics.
• Experience turning customer rubrics into measurable evaluation criteria.
• Hands-on data analytics with Python and the standard data toolkit.

Responsibilities

• Translate customer rubrics into AutoQA evaluation instructions.
• Design and run evaluation experiments on large datasets using LangSmith and BigQuery.
• Curate data into golden datasets and generate synthetic data for edge cases.
• Build LLM-as-a-judge pipelines for repeatable evaluation.
• Make results actionable and collaborate with the AI team to ship improvements.
• Evaluate retrieval, tool calling, and transcription quality across the stack.

Skills

LLM fundamentals

Python

Pandas

NumPy

Jupyter

SQL

Education

Bachelor's degree in a relevant field

Tools

LangSmith

BigQuery

Docker

Kubernetes

Job description

Kaizo Nederland is seeking a sharp data scientist to own evaluation and quality loops for the AutoQA product. You can be early in your career, with room to grow into Senior Data Scientist or ML Engineer roles.

The role sits at the intersection of AI and CX, collaborating with enterprise customers and teams in Amsterdam. We offer flexible hours, visa sponsorship for eligible candidates, and a strong growth path in a fast-growing SaaS company.