AI Evaluation Specialist (Remote, Flexible Projects)

Clera

United States

Remote

USD 41,000 - 83,000

Full time

44 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Payment for hours worked
Fully remote

Job summary

Clera is seeking a detail-oriented professional to design tests that isolate model capabilities and create robust rubrics for scoring. You will develop reference answers, edge cases, and evaluate model outputs against consistent standards to improve benchmarks.

The role emphasizes collaboration, documentation, and trend analysis across experiments. Fully remote and project-based, with work ranging from short studies to longer engagements.

Qualifications

  • Zero to five years of professional experience.
  • QA, test engineering, assessment design, or ML evaluation experience.
  • Experience writing test cases, exam questions, or rubrics.
  • Comfort using spreadsheets or Python to analyze results.
  • Clear written communication.

Responsibilities

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics defining strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating model outputs.

Skills

Test design
Python
Spreadsheets
Clear communication

Tools

Inspect
lm-eval-harness
HELM
promptfoo

Job description

Clera is seeking a detail-oriented professional to design tests that isolate model capabilities and create robust rubrics for scoring. You will develop reference answers, edge cases, and evaluate model outputs against consistent standards to improve benchmarks.

The role emphasizes collaboration, documentation, and trend analysis across experiments. Fully remote and project-based, with work ranging from short studies to longer engagements.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Evaluation & Testing Specialist
Remote AI Evaluation & Testing Specialist

Clera • United States

Remote
USD 41,000 - 83,000
Remote work
Senior AI Evaluation Architect - Remote
Senior AI Evaluation Architect - Remote

Clera • United States

Remote
USD 103,000 - 241,000
Remote Feedback & Alignment Specialist
Remote Feedback & Alignment Specialist

Clera • United States

Remote
USD 28,000 - 62,000
Competitive hourly pay
Fully remote work
Remote AI Evaluation & Feedback Associate
Remote AI Evaluation & Feedback Associate

Clera • United States

Remote
USD 28,000 - 62,000
AI Evaluation Programs Lead (Remote-friendly)
AI Evaluation Programs Lead (Remote-friendly)

CLERA • San Francisco (CA)

On-site
USD 130,000 - 225,000
Visa sponsorship
Relocation assistance
Competitive salary
AI Evaluation Specialist — Remote and Flexible Projects
AI Evaluation Specialist — Remote and Flexible Projects

Synthires • Austin (TX)

Remote
USD 80,000 - 113,000
Competitive pay
Flexible projects
AI Evaluation Specialist - Remote (Engineering Sciences)
AI Evaluation Specialist - Remote (Engineering Sciences)

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
Remote Evaluation Specialist - AI Model Testing & Research
Remote Evaluation Specialist - AI Model Testing & Research

NEPSE Trading • United States

Remote
USD 28,000 - 62,000
AI Evaluation Analyst: Remote, Flexible Projects
AI Evaluation Analyst: Remote, Flexible Projects

Synthires • California

Remote
USD 80,000 - 113,000
Competitive pay
Remote, project-based work
AI Evaluation Specialist (Remote, Part-Time Researcher)
AI Evaluation Specialist (Remote, Part-Time Researcher)

Softechub • United States

Remote
USD 41,000 - 83,000