Data & Evaluations Engineer — Synthetic Data & Metrics

Harvey AI; San Francisco (CA); On-site; USD 230,000 - 450,000; Full time

Job summary

Harvey AI is hiring for a Member of Technical Staff, Data & Evals, to build data and measurement systems powering post-training, benchmarking, and agent capability efforts. You will own synthetic data pipelines, evaluation design, and the systems translating feedback into training signals and rewards.

The role blends research with engineering on a small, high-bar team. You’ll join colleagues who build reliable data and evaluation systems, aligning metrics with real legal work and turning model

Qualifications

• Experience building data pipelines, ML evaluation systems, or experimentation tooling with strong Python skills and ownership of production‑quality code.
• Rigorous approach to measurement, including statistical reasoning and experimental design.
• Ability to translate ambiguous goals or expert judgment into structured datasets, rubrics, and measurable criteria.
• Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus

Responsibilities

• Build synthetic data pipelines that generate, filter, and curate training and evaluation samples for complex legal reasoning and agentic workflows.
• Design evaluations and verifiers that reflect real legal work, calibrate against expert judgment, and investigate failures.
• Build and maintain capability benchmarks like the Legal Agent Benchmark (LAB) and product-focused evals that track model performance on customer workflows.
• Work with legal domain experts to define quality metrics, rubrics, and grading mechanisms for training and evaluation.
• Own training and evaluation data quality, including provenance, coverage, and safeguards against contamination.
• Close the loop from product usage and customer feedback to new evaluations and training data to improve models where needed.
• Partner with researchers to investigate model failures and guide training and releases.

Skills

Data pipelines

ML evaluation

Experimentation tooling

Python

Production-quality code

Statistical measurement

Experimental design

Structured datasets

Rubrics and criteria

Synthetic data

Job description

Harvey AI is hiring for a Member of Technical Staff, Data & Evals, to build data and measurement systems powering post-training, benchmarking, and agent capability efforts. You will own synthetic data pipelines, evaluation design, and the systems translating feedback into training signals and rewards.

The role blends research with engineering on a small, high-bar team. You’ll join colleagues who build reliable data and evaluation systems, aligning metrics with real legal work and turning model