Data Scientist — AutoQA & LLM Evaluation

Slashhash; Amsterdam; Hybrid; EUR 60,000 - 90,000; Full time

Job summary

Kaizo is seeking a Data Scientist to own the evaluation and quality loop of their AutoQA product, which uses LLMs to review customer support conversations. You will translate customer rubrics into testable instructions, run experiments, build datasets and make results actionable.

The ideal candidate has strong Python skills, applied statistics knowledge and 0-2 years of industry experience, with a focus on experimental design and classifier evaluation.

Qualifications

• Strong Python skills and applied statistics knowledge.
• 0-2 years of industry experience.
• Ability to translate customer rubrics into testable instructions and run experiments.

Responsibilities

• Translate customer rubrics into AutoQA instructions.
• Run experiments to improve accuracy using LangSmith and BigQuery.
• Build golden datasets and generate synthetic data.
• Build LLM-as-a-judge pipelines.
• Drive decisions on prompts, tools and capabilities to make results actionable.
• Evaluate retrieval, tool calling, and speech pipelines.
• Join customer calls to understand quality definitions.

Skills

Python

Pandas

NumPy

Jupyter

Applied Statistics

Experiment Design

Classifier Evaluation

Precision/Recall Trade-offs

Imbalanced Data

LLMs

Prompt Engineering

LangSmith

BigQuery

SQL

Tools

Docker

Kubernetes

Google Cloud Platform

Job description

Kaizo is seeking a Data Scientist to own the evaluation and quality loop of their AutoQA product, which uses LLMs to review customer support conversations. You will translate customer rubrics into testable instructions, run experiments, build datasets and make results actionable.

The ideal candidate has strong Python skills, applied statistics knowledge and 0-2 years of industry experience, with a focus on experimental design and classifier evaluation.