Senior Evaluation Operations Lead — Data Quality

Harvey; California (MO); On-site; USD 155,000 - 233,000; Full time

Job summary

Harvey is seeking a senior operator to own the quality bar for human evaluations as it scales globally. You will define what good looks like, author guidelines, and build repeatable evaluation templates while partnering with ALR, Product, Engineering, and AI Research to ship trusted outputs.

You will lead calibration, inter-rater reliability, and analytic workflows to diagnose issues and deliver actionable signals for product decisions.

Qualifications

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
• Experience owning quality inputs for complex, expert-driven work with repeatable standards.
• Onboarding, training, calibrating a distributed pool of raters, annotators, or reviewers.

Responsibilities

• Own quality inputs and define 'good' for eval methodology and data analysis.
• Author and maintain evaluation guidelines, instructions, and databases.
• Standardize rubric design into repeatable templates with documented methodology.
• Manage onboarding, calibration training, inter-rater reliability, and feedback loops.
• Conduct quantitative and qualitative analyses, diagnose errors, and surface actionable signals.
• Run QA on vendor deliverables before launch decisions.
• Ensure quality across jurisdictions and non-English geographies.
• Develop dashboards to track rater capability and program health.
• Support ALR in review steps to certify evaluation soundness.
• Collaborate to align human eval with online signals and insights.

Skills

Product operations

Research operations

QA operations

Quality program management

Tools

Calibration methods

Inter-rater reliability

Statistical tests

Job description

Harvey is seeking a senior operator to own the quality bar for human evaluations as it scales globally. You will define what good looks like, author guidelines, and build repeatable evaluation templates while partnering with ALR, Product, Engineering, and AI Research to ship trusted outputs.

You will lead calibration, inter-rater reliability, and analytic workflows to diagnose issues and deliver actionable signals for product decisions.