Senior Evaluation Expert (Fully Remote)

Surge AI

United States

Remote

USD 90,000 - 150,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Surge AI is seeking a senior evaluator to lead benchmark design and validation for AI research projects in the United States. You will guide grading operations, calibrate graders, and resolve disputes while shaping scoring methods for standardized benchmarks.

Ideal candidates bring 5+ years in evaluation, and proven ability to design assessments and develop gold-standard answers. Proficiency in Python and evaluation tools is required to drive rigorous benchmarking programs.

Qualifications

  • 5+ years of experience in evaluation or domain expertise.
  • Experience designing assessments or benchmarks used by others.
  • Proficiency in Python and evaluation tools.

Responsibilities

  • Lead benchmark design and validation for AI research projects.
  • Guide grading operations including calibrating graders and resolving disputes.
  • Design scope, question types, difficulty levels, and scoring methods for benchmarks.
  • Write and review rubrics to ensure consistent grading.
  • Develop gold-standard answers and test sets.

Skills

Python
Evaluation tools
Benchmark design

Tools

Evaluation frameworks

Job description

  • Lead benchmark design and validation for AI research projects.
  • Guide grading operations including calibrating graders and resolving disputes.
Key Responsibilities:
  • Design scope, question types, difficulty levels, and scoring methods for benchmarks.
  • Write and review rubrics to ensure consistent grading.
  • Develop gold-standard answers and test sets.
Qualifications:
  • At least 5 years of experience in evaluation or domain expertise.
  • Experience designing assessments or benchmarks used by others.
  • Proficiency in Python and evaluation tools.
Surge AI

Surge AI connects senior evaluation experts to support AI research and model testing. The company focuses on rigorous benchmark design and grading for AI systems, with a network of specialized contractors working on diverse projects.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Evaluation Architect for AI Benchmarks
Senior Evaluation Architect for AI Benchmarks

Surge AI • United States

Remote
USD 90,000 - 150,000
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera Labs, Inc. • United States

Remote
USD 103,000 - 241,000
Senior AI Evaluation Architect - Remote
Senior AI Evaluation Architect - Remote

Clera Labs, Inc. • United States

Remote
USD 103,000 - 241,000
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • United States

Remote
USD 90,000 - 150,000
Remote Scientific & Technical Evaluation Specialist
Remote Scientific & Technical Evaluation Specialist

Meridial • United States

Remote
USD 14,000 - 41,000
Expert Talent Pool, Evals and Benchmarks (Associate)
Expert Talent Pool, Evals and Benchmarks (Associate)

Clera Labs, Inc. • United States

Remote
USD 41,000 - 83,000
Remote work
Scientific and Technical Service Specialist - Freelance AI Trainer Project
Scientific and Technical Service Specialist - Freelance AI Trainer Project

Meridial • United States

Remote
USD 14,000 - 41,000
Scientific and Technical Service Specialist - Freelance AI Trainer Project
Scientific and Technical Service Specialist - Freelance AI Trainer Project

Invisible Technologies Inc. • United States

Remote
USD 14,000 - 41,000
Remote AI Benchmark Researcher - Computational STEM
Remote AI Benchmark Researcher - Computational STEM

Weekday • United States

Remote
USD 83,000 - 124,000
Information Specialist - Freelance AI Trainer Project
Information Specialist - Freelance AI Trainer Project

Meridial • United States

Remote
USD 14,000 - 41,000