Senior AI Evaluation Engineer — Frontier Models

Black Forest Labs; Zürich; Hybrid; CHF 139,000 - 203,000; Full time

Job summary

Black Forest Labs in Freiburg or Zurich seeks a Senior Evaluation Engineer to build and maintain the evaluation stack for FLUX models, including automated benchmarking and LLM-as-judge pipelines. You will lead human panels and refine metrics for multimodal evaluation across image, video, and beyond.

The role requires deep expertise in Python, PyTorch, and statistics, plus experience with benchmarking and ML infrastructure. Collaboration with RLHF researchers is essential to improve checkpoints.

Qualifications

• Advanced degree (MS/PhD) in a relevant field or equivalent practical experience.
• Practical experience evaluating generative models with hands-on work.
• Strong Python and software engineering skills for reliable pipelines.
• Working knowledge of PyTorch and ML infrastructure practices.
• Solid grasp of statistics, sample sizes, and inter-annotator agreement.
• Experience building or operating LLM-as-judge systems.
• Familiarity with benchmarking datasets and standard benchmarks.

Responsibilities

• Build and maintain the evaluation stack that every model checkpoint passes through.
• Automate benchmarking and LLM-as-judge pipelines and manage human panels.
• Design broad evaluation across taxonomy: prompt adherence, aesthetics, safety.
• Ensure eval results remain stable over time and improve them when needed.
• Benchmark FLUX against frontier models and interpret differences meaningfully.
• Identify outputs where quality falls short not captured by aggregates.
• Collaborate with RL and finetuning researchers to set target metrics.

Skills

Python

Research evaluation

Data analysis

ML pipelines

Education

MS/PhD in a relevant field

Tools

PyTorch

Prolific

Job description

Black Forest Labs in Freiburg or Zurich seeks a Senior Evaluation Engineer to build and maintain the evaluation stack for FLUX models, including automated benchmarking and LLM-as-judge pipelines. You will lead human panels and refine metrics for multimodal evaluation across image, video, and beyond.

The role requires deep expertise in Python, PyTorch, and statistics, plus experience with benchmarking and ML infrastructure. Collaboration with RLHF researchers is essential to improve checkpoints.