Expert Talent Pool, Evals and Benchmarks

Clera

United States

Remote

USD 103,000 - 241,000

Full time

44 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Clera Labs, Inc. is seeking senior evaluation experts to lead benchmark design, validation, and grading for AI research projects. You will design benchmarks, write rubrics, develop gold-standard answers, and supervise grader teams.

This fully remote role offers compensation per hour from $75 to $175, depending on project and experience, with assignments ranging from one week to several months. Excellent English communication and ML evaluation background required.

Qualifications

  • 5+ years of experience in evaluation, testing, or assessment.
  • Experience designing a benchmark, test plan, exam, or assessment used by others.
  • Proficiency in Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.
  • Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.
  • Clear written English communication.

Responsibilities

  • Design benchmarks, including scope, question types, difficulty levels, and scoring methods.
  • Write and review rubrics to support consistent grading.
  • Develop gold-standard answers and test sets that hold up to expert review.
  • Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.
  • Analyze model failures, identify root causes, and recommend improvements.
  • Design and conduct rigorous model comparison studies.
  • Validate benchmarks for contamination, ambiguity, and saturation.
  • Design user studies on how people rate and interact with model outputs.

Skills

Benchmark design
Rubrics writing
Gold-standard answers
Grading calibration
Model evaluation
User studies design
Designing assessments
Strong written English

Tools

Inspect
lm-eval-harness
HELM
promptfoo
Python

Job description

About the Role

Join a network of senior evaluation experts supporting AI research projects. You will lead benchmark design, validation, and grading operations when projects align with your expertise, helping produce rigorous and reliable assessments of AI models.

What You'll Do
  • Design benchmarks, including their scope, question types, difficulty levels, and scoring methods.

  • Write and review rubrics to support consistent grading.

  • Develop gold-standard answers and test sets that hold up to expert review.

  • Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.

  • Analyze model failures, identify root causes, and recommend improvements.

  • Design and conduct rigorous model comparison studies.

  • Validate benchmarks for contamination, ambiguity, and saturation.

  • Design user studies on how people rate and interact with model outputs.

What We're Looking For
  • At least 5 years of experience in evaluation, testing, or assessment, or at least 5 years of domain expertise combined with hands-on evaluation work.

  • Experience designing a benchmark, test plan, exam, or assessment used by others.

  • Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.

  • Proficiency in Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.

  • Experience analyzing model or test failures and recommending fixes.

  • Background in ML evaluation, QA, test engineering, psychometrics, assessment design, UX research, data science, or deep subject matter expertise.

  • Clear written English communication.

  • Experience validating benchmarks, measuring grader agreement, conducting model comparisons, or running user studies is relevant.

Compensation & Benefits

Compensation is $75 to $175 per hour, depending on project, field, and experience level. Pay is based on hours worked.

Location

Fully remote. Projects may last from one week to several months.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • United States

Remote
USD 41,000 - 83,000
Remote work
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • United States

Remote
USD 90,000 - 150,000
Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • United States

Remote
USD 41,000 - 83,000
Payment for hours worked
Fully remote
Senior AI Evaluation Architect - Remote
Senior AI Evaluation Architect - Remote

Clera • United States

Remote
USD 103,000 - 241,000
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)

Clera • United States

Remote
USD 69,000 - 165,000
Expert Talent Pool, Feedback and Alignment Associate
Expert Talent Pool, Feedback and Alignment Associate

Clera • United States

Remote
USD 28,000 - 62,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Bake AI Inc. • San Mateo (CA)

On-site
USD 180,000 - 240,000
Scientific and Technical Service Specialist
Scientific and Technical Service Specialist

Meridial • United States

Remote
USD 14,000 - 41,000
Machine Learning Engineer, AI Evaluation & Benchmarks
Machine Learning Engineer, AI Evaluation & Benchmarks

Vettedlance • United States

Remote
USD 83,000 - 165,000
Remote work
Hourly independent contractor
Part-time flexible 10–40 hrs/week
+1
Scientific and Technical Service Specialist - Freelance AI Trainer Project
Scientific and Technical Service Specialist - Freelance AI Trainer Project

Invisible Technologies Inc. • United States

Remote
USD 14,000 - 41,000