Expert Talent Pool, Evals and Benchmarks (Associate)

Clera

United States

A distancia

USD 41.000 - 83.000

Jornada completa

hace 44 horas
Sé de los primeros/as/es en solicitar esta vacante

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Ventajas ofrecidas por este puesto de trabajo

Payment for hours worked
Fully remote

Descripción de la vacante

Clera is seeking a detail-oriented professional to design tests that isolate model capabilities and create robust rubrics for scoring. You will develop reference answers, edge cases, and evaluate model outputs against consistent standards to improve benchmarks.

The role emphasizes collaboration, documentation, and trend analysis across experiments. Fully remote and project-based, with work ranging from short studies to longer engagements.

Formación

  • Zero to five years of professional experience.
  • QA, test engineering, assessment design, or ML evaluation experience.
  • Experience writing test cases, exam questions, or rubrics.
  • Comfort using spreadsheets or Python to analyze results.
  • Clear written communication.

Responsabilidades

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics defining strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating model outputs.

Conocimientos

Test design
Python
Spreadsheets
Clear communication

Herramientas

Inspect
lm-eval-harness
HELM
promptfoo

Descripción del empleo

Responsibilities
  • Write questions and tasks that isolate and test specific model capabilities
  • Develop rubrics defining strong, partial, and poor answers
  • Create reference answers and edge cases to challenge AI models
  • Grade model outputs against rubrics and explain scoring decisions
  • Analyze failed answers, group error patterns, and document findings
  • Compare multiple models using identical prompts
  • Validate benchmarks and flag compromised or ineffective questions
  • Participate in small user studies evaluating model outputs
Requirements
  • Zero to five years of professional experience
  • QA, test engineering, assessment design, or ML evaluation experience
  • Experience writing test cases, exam questions, or rubrics
  • Comfort using spreadsheets or Python to analyze results
  • Clear written communication
Preferred Qualifications
  • Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo
Benefits
  • Compensation of $30 to $60 per hour, depending on project, field, and experience
  • Payment for hours worked
  • Fully remote; projects typically last one week to several months
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • EE. UU.

A distancia
USD 41.000 - 83.000
Remote work
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • EE. UU.

A distancia
USD 103.000 - 241.000
Expert Talent Pool, Feedback and Alignment Associate
Expert Talent Pool, Feedback and Alignment Associate

Clera • EE. UU.

A distancia
USD 28.000 - 62.000
Competitive hourly pay
Fully remote work
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • EE. UU.

A distancia
USD 90.000 - 150.000
Expert Talent Pool, Feedback and Alignment Associate
Expert Talent Pool, Feedback and Alignment Associate

Clera • EE. UU.

A distancia
USD 28.000 - 62.000
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)
Expert Talent Pool, Feedback and Alignment (Senior, 5+ Years)

Clera • EE. UU.

A distancia
USD 69.000 - 165.000
Researcher | $60/hr Remote
Researcher | $60/hr Remote

Crossing Hurdles • EE. UU.

A distancia
USD 28.000 - 83.000
AI QA Trainer
AI QA Trainer

Meridial • EE. UU.

A distancia
USD 8300 - 90.000
AI QA Trainer - LLM Evaluation
AI QA Trainer - LLM Evaluation

Meridial • EE. UU.

A distancia
USD 8300 - 90.000
AI QA Trainer - LLM Evaluation - Freelance Project
AI QA Trainer - LLM Evaluation - Freelance Project

Meridial • EE. UU.

A distancia
USD 8265 - 89.544
Secure computer and high-speed internet required