Senior AI Evaluation Architect - Remote

Clera

United States

Remote

USD 103,000 - 241,000

Full time

44 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Clera Labs, Inc. is seeking senior evaluation experts to lead benchmark design, validation, and grading for AI research projects. You will design benchmarks, write rubrics, develop gold-standard answers, and supervise grader teams.

This fully remote role offers compensation per hour from $75 to $175, depending on project and experience, with assignments ranging from one week to several months. Excellent English communication and ML evaluation background required.

Qualifications

  • 5+ years of experience in evaluation, testing, or assessment.
  • Experience designing a benchmark, test plan, exam, or assessment used by others.
  • Proficiency in Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.
  • Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.
  • Clear written English communication.

Responsibilities

  • Design benchmarks, including scope, question types, difficulty levels, and scoring methods.
  • Write and review rubrics to support consistent grading.
  • Develop gold-standard answers and test sets that hold up to expert review.
  • Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.
  • Analyze model failures, identify root causes, and recommend improvements.
  • Design and conduct rigorous model comparison studies.
  • Validate benchmarks for contamination, ambiguity, and saturation.
  • Design user studies on how people rate and interact with model outputs.

Skills

Benchmark design
Rubrics writing
Gold-standard answers
Grading calibration
Model evaluation
User studies design
Designing assessments
Strong written English

Tools

Inspect
lm-eval-harness
HELM
promptfoo
Python

Job description

Clera Labs, Inc. is seeking senior evaluation experts to lead benchmark design, validation, and grading for AI research projects. You will design benchmarks, write rubrics, develop gold-standard answers, and supervise grader teams.

This fully remote role offers compensation per hour from $75 to $175, depending on project and experience, with assignments ranging from one week to several months. Excellent English communication and ML evaluation background required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • United States

Remote
USD 103,000 - 241,000
Remote AI Evaluation & Testing Specialist
Remote AI Evaluation & Testing Specialist

Clera • United States

Remote
USD 41,000 - 83,000
Remote work
Senior AI Quality Grader & Rater - Remote
Senior AI Quality Grader & Rater - Remote

Clera • United States

Remote
USD 69,000 - 165,000
AI Evaluation Specialist (Remote, Flexible Projects)
AI Evaluation Specialist (Remote, Flexible Projects)

Clera • United States

Remote
USD 41,000 - 83,000
Payment for hours worked
Fully remote
AI Evaluation Analyst: Remote, Flexible Projects
AI Evaluation Analyst: Remote, Flexible Projects

Synthires • California

Remote
USD 80,000 - 113,000
Competitive pay
Remote, project-based work
Generalist Expert (Remote | $50–$70/hr)
Generalist Expert (Remote | $50–$70/hr)

Synthires • United States

Remote
USD 80,000 - 113,000
Generalist Expert (Remote | $50–$70/hr)
Generalist Expert (Remote | $50–$70/hr)

Synthires • California

Remote
USD 80,000 - 113,000
AI Evaluation & Research Associate - Remote & Flexible Hours
AI Evaluation & Research Associate - Remote & Flexible Hours

prolificacademicltd • United States

Remote
USD 68,000 - 97,000
Remote AI Evaluation & Feedback Associate
Remote AI Evaluation & Feedback Associate

Clera • United States

Remote
USD 28,000 - 62,000
Remote AI Evaluation Scientist | Flexible Hours
Remote AI Evaluation Scientist | Flexible Hours

prolificacademicltd • United States

Remote
USD 68,000 - 97,000