Research Engineer - AI Taste & Evaluation Pipelines

Taste Labs; San Francisco (CA); On-site; USD 140,000 - 220,000; Full time

Job summary

Taste Labs in San Francisco is seeking a versatile ML engineer to own end-to-end experimentation pipelines—from training runs to evaluation and post-training infra. You’ll publish results and collaborate with AI labs on frontier projects.

You’ll build reward models, judges and verification tools for subjective tasks like design, writing, and visual style; you’ll run post-training experiments on open-source models and shape Taste’s evaluator stack.

Qualifications

• Experience with LLMs and evaluation of subjective domains.
• Ability to design and run post-training experiments.
• Experience building end-to-end ML pipelines and evaluators.
• Familiarity with research-style thinking in a startup setting.

Responsibilities

• Train reward models, classifiers, judges, and verifiers for subjective domains (e.g. design, writing, visual style).
• Develop frontier evaluations and benchmarks for subjective domains.
• Run post-training experiments on open-source models to test new data formats and post-training techniques.
• Setup the infrastructure to run this.
• Collaborate with AI labs and creative experts to design pilots and experiments around taste.
• Own the end to end pipeline.
• Publish blogs and whitepapers.

Skills

LLM experience

Post-training

Eval/Judges

Full-stack ML

Infra setup

Research mindset

Creative flair

Tools

Python

PyTorch

TensorFlow

Open-source models

Job description

Taste Labs in San Francisco is seeking a versatile ML engineer to own end-to-end experimentation pipelines—from training runs to evaluation and post-training infra. You’ll publish results and collaborate with AI labs on frontier projects.

You’ll build reward models, judges and verification tools for subjective tasks like design, writing, and visual style; you’ll run post-training experiments on open-source models and shape Taste’s evaluator stack.