Expert Talent Pool, Evals and Benchmarks

Clera

United States

A distancia

USD 90.000 - 150.000

Jornada completa

hace 44 horas
Sé de los primeros/as/es en solicitar esta vacante

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

The network supports AI research projects with rigorous model evaluation.

We are seeking an experienced evaluator to design benchmarks, write rubrics and gold-standard answers, lead grading rounds, analyze model failures, and run comparative studies to drive improvements. Strong English communication and Python proficiency are essential for success in this role.

Formación

  • 5+ years in evaluation, testing or assessment with relevant domain expertise.
  • Experience designing a benchmark, test plan, exam or assessment.
  • Experience writing or reviewing rubrics and gold-standard answers.
  • Proficiency in Python and evaluation tools.
  • Clear written English communication.

Responsabilidades

  • Design benchmarks covering scope, question types, difficulty and scoring
  • Write and review rubrics and gold-standard answers
  • Lead grading rounds, calibrate graders and resolve disputes
  • Analyze model failures and recommend improvements
  • Run model comparison studies and validate benchmarks
  • Design user studies on how people rate model outputs

Conocimientos

Clear written English communication

Educación

5+ years in evaluation, testing or assessment
Experience designing a benchmark, test plan, exam or assessment

Herramientas

Python
Evaluation tools

Descripción del empleo

Responsibilities
  • Design benchmarks covering scope, question types, difficulty and scoring
  • Write and review rubrics and gold-standard answers
  • Lead grading rounds, calibrate graders and resolve disputes
  • Analyze model failures and recommend improvements
  • Run model comparison studies and validate benchmarks
  • Design user studies on how people rate model outputs
Requirements
  • 5+ years in evaluation, testing or assessment, or domain expertise with evaluation work
  • Experience designing a benchmark, test plan, exam or assessment
  • Experience writing or reviewing rubrics and gold-standard answers
  • Proficiency in Python and evaluation tools
  • Clear written English communication
Preferred Qualifications
  • Background in ML evaluation, QA, psychometrics, UX research or data science
  • Experience validating benchmarks or measuring grader agreement
About the Company

The network supports AI research projects with rigorous model evaluation.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • EE. UU.

A distancia
USD 103.000 - 241.000
Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • EE. UU.

A distancia
USD 41.000 - 83.000
Payment for hours worked
Fully remote
Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera • EE. UU.

A distancia
USD 41.000 - 83.000
Remote work
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Bake AI Inc. • San Mateo (CA)

Presencial
USD 180.000 - 240.000
LLM Benchmarking Research Scientist
LLM Benchmarking Research Scientist

Anyone AI Inc. • EE. UU.

Híbrido
USD 110.000 - 160.000
Research Scientist
Research Scientist

Anyone AI Labs • EE. UU.

A distancia
USD 120.000 - 180.000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • EE. UU.

A distancia
USD 120.000 - 180.000
Member of Technical Staff, Model Evaluation
Member of Technical Staff, Model Evaluation

Inception Labs • Palo Alto (CA)

Híbrido
USD 150.000 - 185.000
Research Scientist
Research Scientist

Anyone AI Inc. • EE. UU.

A distancia
USD 110.000 - 160.000
AI Engineer (Data and Evaluations)
AI Engineer (Data and Evaluations)

Glean • San Francisco (CA)

Presencial
USD 150.000 - 210.000