Expert Talent Pool, Evals and Benchmarks (Associate)

Clera

United States

A distancia

USD 41.000 - 83.000

Jornada completa

hace 44 horas
Sé de los primeros/as/es en solicitar esta vacante

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Ventajas ofrecidas por este puesto de trabajo

Remote work

Descripción de la vacante

Clera Labs, Inc. is seeking a remote QA-focused role to design and evaluate assessments for AI model safety and capabilities. You will write test prompts, develop scoring rubrics, and create reference answers to challenge models across varied tasks.

The role involves grading outputs, analyzing error patterns, and comparing models using identical prompts, with emphasis on clear documentation and communication.

Formación

  • Zero to five years of professional experience.
  • Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing.
  • Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
  • Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns.

Responsabilidades

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating ratings or interactions with model outputs.

Conocimientos

QA
Test design
Psychometrics
UX research
Data analysis
ML evaluation
Research support
Question writing

Herramientas

Inspect
lm-eval-harness
HELM
promptfoo

Descripción del empleo

About the Role

Join a specialist talent pool supporting human-data projects that evaluate AI models. You will help design and run assessments that measure model capabilities and contribute to AI safety and capability research.

What You'll Do
  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating ratings or interactions with model outputs.
What We're Looking For
  • Zero to five years of professional experience.
  • Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing.
  • Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
  • Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns.
  • Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo is useful.
  • Clear written communication and the ability to compare AI model outputs or participate in user and rating studies.
Compensation & Benefits

Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked.

Location

Fully remote. Projects typically last from one week to several months.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • EE. UU.

A distancia
USD 41.000 - 83.000
Payment for hours worked
Fully remote
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • EE. UU.

A distancia
USD 103.000 - 241.000
Expert Talent Pool, Feedback and Alignment Associate
Expert Talent Pool, Feedback and Alignment Associate

Clera • EE. UU.

A distancia
USD 28.000 - 62.000
Expert Talent Pool, Feedback and Alignment Associate
Expert Talent Pool, Feedback and Alignment Associate

Clera • EE. UU.

A distancia
USD 28.000 - 62.000
Competitive hourly pay
Fully remote work
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • EE. UU.

A distancia
USD 90.000 - 150.000
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)

Clera • EE. UU.

A distancia
USD 69.000 - 165.000
Expert Talent Pool, Generation and Writing Associate
Expert Talent Pool, Generation and Writing Associate

Clera • EE. UU.

A distancia
USD 34.000 - 69.000
Fully remote
Expert Talent Pool, Generation and Writing (Senior)
Expert Talent Pool, Generation and Writing (Senior)

Clera • EE. UU.

A distancia
USD 83.000 - 207.000
Expert Talent Pool, Safety Testing (Expert)
Expert Talent Pool, Safety Testing (Expert)

Clera • EE. UU.

A distancia
USD 103.000 - 241.000
Fully remote
Expert Talent Pool, Generation and Writing (Senior)
Expert Talent Pool, Generation and Writing (Senior)

Clera • EE. UU.

A distancia
USD 83.000 - 207.000
Hourly pay: $60-$150
Remote work
AI training project exposure