Staff Data & Evaluations Engineer, Legal AI Post-Training

Harvey AI; New York (NY); On-site; USD 230,000 - 450,000; Full time

Job summary

Harvey Research seeks a Member of Technical Staff, Data & Evals to build data and measurement systems powering post-training, benchmarking, and agent capability efforts. You’ll own synthetic data pipelines, evaluation design, and grading frameworks for training environments.

You’ll join a small, high-bar team of engineers and researchers, owning a substantial portion of the work and turning feedback into measurable improvements for real-world legal workflows.

Qualifications

• Experience building data pipelines and ML evaluation systems with production‑quality code.
• Rigorous measurement approach: statistical reasoning, experimental design, and awareness of metric pitfalls.
• Ability to translate vague goals or expert judgments into structured datasets and rubrics.
• Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.

Responsibilities

• Build synthetic data pipelines for training and evaluation samples in legal reasoning contexts.
• Design evaluations and verifiers calibrated to expert judgment and investigate failures.
• Maintain capability benchmarks like the Legal Agent Benchmark (LAB) and product-focused evals.
• Define quality metrics and rubrics with legal domain experts for training and evaluation.
• Own data quality, provenance, coverage and safeguards against contamination.
• Close loop from product usage to new evaluations and training data for model improvement.
• Collaborate with researchers to analyze model failures and guide training and releases.

Skills

Data pipelines, ML evaluation systems

Measurement and experimental design

Structured datasets and rubrics

ML model evaluation insights

Tools

Python

Job description

Harvey Research seeks a Member of Technical Staff, Data & Evals to build data and measurement systems powering post-training, benchmarking, and agent capability efforts. You’ll own synthetic data pipelines, evaluation design, and grading frameworks for training environments.

You’ll join a small, high-bar team of engineers and researchers, owning a substantial portion of the work and turning feedback into measurable improvements for real-world legal workflows.