Legal Rubric & Ground Truth Auditor (Licensed Attorney)

Confidential

United States

Remote

USD 83,000 - 165,000

Part time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Confidential is seeking licensed attorneys to audit rubrics and ground-truth answers for a legal research dataset used to evaluate advanced AI systems. You will review existing questions, verify model answers against controlling statutes, regulations, and case law, and refine grading rubrics for accuracy and clarity.

This short-term, fully remote engagement requires U.S. residency and strong legal research, writing, and analytical skills across multiple practice areas, including Administrative,

Qualifications

  • JD and active U.S. bar admission in good standing required.
  • 5+ years of legal practice with substantive experience in one or more practice areas listed.
  • Strong research, analytical, and writing abilities essential.
  • High attention to detail and ability to apply consistent judgment.

Responsibilities

  • Audit grading rubrics for accuracy, completeness, consistency, and clarity.
  • Verify ground-truth answers against statutes, regulations, and case law.
  • Identify legal errors, outdated authority, missing issues, and ambiguous criteria.
  • Rewrite or correct ground-truth answers and rubric items as needed.
  • Provide concise, structured written feedback on each item reviewed.

Skills

5+ years of legal practice
Strong legal research, analysis, and -
High attention to detail and judgment

Education

Juris Doctor (JD) and active U.S. bar admission in good standing

Job description

our partner is seeking experienced licensed attorneys to audit rubrics and ground-truth answers for a legal research dataset used to evaluate advanced AI systems. You will review existing legal research questions, verify the accuracy of model answers, and refine grading rubrics so the dataset reflects current law and sound legal reasoning. This is a short-term, fully remote engagement.

Responsibilities
  • Audit grading rubrics for accuracy, completeness, consistency, and clarity.

  • Verify ground-truth answers against controlling statutes, regulations, and case law.

  • Identify legal errors, outdated authority, missing issues, and ambiguous grading criteria.

  • Rewrite or correct ground-truth answers and rubric items where needed.

  • Provide concise, structured written feedback on each item reviewed.

Practice Areas

The dataset spans eight practice areas. Experience in one or more is required, and breadth across several is a plus:

  • Administrative / Regulatory

  • Business & Commercial

  • Civil Litigation

  • Constitutional / Civil Rights

  • Criminal

  • Family

  • Health

  • Immigration

Required Qualifications
  • Juris Doctor (JD) and active U.S. bar admission in good standing.

  • 5+ years of legal practice.

  • Substantive experience in at least one of the practice areas above.

  • Strong legal research, analysis, and writing skills.

  • High attention to detail and the ability to apply consistent judgment.

Preferred Qualifications
  • Experience across multiple listed practice areas.

  • Judicial clerkship, appellate, government, or law firm research experience.

  • Prior experience reviewing or grading legal work product, or with AI / legal-tech evaluation projects.

Why Join?
  • Apply your legal expertise to shape how AI systems are evaluated for legal research.

  • Collaborate with a leading AI research organisation.

This is a 100% remote contract role. Applicants must be located in: United States.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Legal Rubric & Ground Truth Auditor (Attorney) · Mercor Mercor · $120-140/hr · remote in US · 3d ago $120-140/hr remote in US 3d ago
Legal Rubric & Ground Truth Auditor (Attorney) · Mercor Mercor · $120-140/hr · remote in US · 3d ago $120-140/hr remote in US 3d ago

Benture • United States

Remote
USD 165,000 - 193,000
Legal Rubric & Ground Truth Auditor (Licensed Attorney)
Legal Rubric & Ground Truth Auditor (Licensed Attorney)

Mercor • United States

Remote
USD 83,000 - 124,000
Remote Legal Rubric & Ground-Truth Auditor
Remote Legal Rubric & Ground-Truth Auditor

Mercor • United States

Remote
USD 83,000 - 124,000
Remote Legal Rubric & Ground-Truth Auditor (JD, 5+ yrs)
Remote Legal Rubric & Ground-Truth Auditor (JD, 5+ yrs)

Benture • United States

Remote
USD 165,000 - 193,000
Remote Legal Rubric Auditor — Licensed Attorney
Remote Legal Rubric Auditor — Licensed Attorney

Confidential • United States

Remote
USD 83,000 - 165,000
AI Expert Contributor
AI Expert Contributor

Consilio LLC • United States

Remote
USD 62,000 - 76,000
Remote work
Part-time hours
ASAP start
Remote AI Legal Reviewer (Contractor)
Remote AI Legal Reviewer (Contractor)

Weekday • United States

Remote
USD 138,000 - 234,000
Remote Legal Expert - 42119
Remote Legal Expert - 42119

Turing • Chicago (IL)

Remote
USD 55,000 - 110,000
Flexible hours
Contractor arrangement
Legal & Compliance Specialist – Rubric Design
Legal & Compliance Specialist – Rubric Design

SME Careers • United States

Remote
USD 55,000 - 96,000
Lawyer
Lawyer

Remolity, Inc. • United States

Remote
USD 110,000 - 165,000