AI Evaluation Engineer — Data Pipelines & Benchmarks

Harvey; United States; Remote; USD 230,000 - 450,000; Full time

Job summary

Harvey AI is seeking a research engineer to build data and evaluation systems powering post-training, benchmarking, and agent capability efforts in legal AI. You’ll own synthetic data pipelines and evaluation design, turning domain expert feedback into training signals.

Join a high-bar team of engineers and researchers, contribute to measurements that matter, and help translate complex legal work into robust, production-grade data and metrics that scale with our growing platform.

Qualifications

• Experience building data pipelines and ML evaluation systems.
• Strong Python skills and ownership of production-quality code.
• Ability to translate ambiguous goals into structured datasets and rubrics.
• Experience with synthetic data or LLM evaluations is a plus.

Responsibilities

• Build synthetic data pipelines to generate, filter, and curate training and evaluation samples for complex legal reasoning.
• Design evaluations and verifiers reflecting real legal work, calibrate to expert judgment, investigate failures.
• Build and maintain capability benchmarks like LAB and product-focused evals tracking model performance on real customer workflows.
• Collaborate with legal experts to define quality metrics, rubrics, and grading mechanisms.
• Own training and evaluation data quality, including provenance, coverage, and contamination safeguards.
• Close the loop from usage and feedback to new evaluations and training data to improve models where needed.
• Partner with researchers to investigate failures and guide training decisions.

Skills

Python

Experiment design

Data pipelines

Production-grade code

Job description

Harvey AI is seeking a research engineer to build data and evaluation systems powering post-training, benchmarking, and agent capability efforts in legal AI. You’ll own synthetic data pipelines and evaluation design, turning domain expert feedback into training signals.

Join a high-bar team of engineers and researchers, contribute to measurements that matter, and help translate complex legal work into robust, production-grade data and metrics that scale with our growing platform.