Remote AI Evaluation & Testing Specialist

Clera

United States

Remote

USD 41,000 - 83,000

Full time

44 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote work

Job summary

Clera Labs, Inc. is seeking a remote QA-focused role to design and evaluate assessments for AI model safety and capabilities. You will write test prompts, develop scoring rubrics, and create reference answers to challenge models across varied tasks.

The role involves grading outputs, analyzing error patterns, and comparing models using identical prompts, with emphasis on clear documentation and communication.

Qualifications

  • Zero to five years of professional experience.
  • Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing.
  • Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
  • Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns.

Responsibilities

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating ratings or interactions with model outputs.

Skills

QA
Test design
Psychometrics
UX research
Data analysis
ML evaluation
Research support
Question writing

Tools

Inspect
lm-eval-harness
HELM
promptfoo

Job description

Clera Labs, Inc. is seeking a remote QA-focused role to design and evaluate assessments for AI model safety and capabilities. You will write test prompts, develop scoring rubrics, and create reference answers to challenge models across varied tasks.

The role involves grading outputs, analyzing error patterns, and comparing models using identical prompts, with emphasis on clear documentation and communication.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Safety Testing Specialist
Remote AI Safety Testing Specialist

Clera • United States

Remote
USD 41,000 - 83,000
Remote AI Evaluation & Feedback Associate
Remote AI Evaluation & Feedback Associate

Clera • United States

Remote
USD 28,000 - 62,000
Senior AI Safety Tester & Red Team Lead (Remote)
Senior AI Safety Tester & Red Team Lead (Remote)

Clera • United States

Remote
USD 103,000 - 241,000
Fully remote
AI Evaluation Specialist (Remote, Flexible Projects)
AI Evaluation Specialist (Remote, Flexible Projects)

Clera • United States

Remote
USD 41,000 - 83,000
Payment for hours worked
Fully remote
Remote Evaluation Specialist - AI Model Testing & Research
Remote Evaluation Specialist - AI Model Testing & Research

NEPSE Trading • United States

Remote
USD 28,000 - 62,000
Senior AI Quality Grader & Rater - Remote
Senior AI Quality Grader & Rater - Remote

Clera • United States

Remote
USD 69,000 - 165,000
Senior AI Evaluation Architect - Remote
Senior AI Evaluation Architect - Remote

Clera • United States

Remote
USD 103,000 - 241,000
Remote QA Engineer for AI Data Validation & Evaluation
Remote QA Engineer for AI Data Validation & Evaluation

YO AI Labs • United States

Remote
USD 55,000 - 96,000
Remote Feedback & Alignment Specialist
Remote Feedback & Alignment Specialist

Clera • United States

Remote
USD 28,000 - 62,000
Competitive hourly pay
Fully remote work
Remote AI Evaluation Specialist
Remote AI Evaluation Specialist

Anyone AI Inc. • United States

Remote
USD 28,000 - 48,000