Senior Evaluation Architect for AI Benchmarks

Surge AI

United States

Remote

USD 90,000 - 150,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Surge AI is seeking a senior evaluator to lead benchmark design and validation for AI research projects in the United States. You will guide grading operations, calibrate graders, and resolve disputes while shaping scoring methods for standardized benchmarks.

Ideal candidates bring 5+ years in evaluation, and proven ability to design assessments and develop gold-standard answers. Proficiency in Python and evaluation tools is required to drive rigorous benchmarking programs.

Qualifications

  • 5+ years of experience in evaluation or domain expertise.
  • Experience designing assessments or benchmarks used by others.
  • Proficiency in Python and evaluation tools.

Responsibilities

  • Lead benchmark design and validation for AI research projects.
  • Guide grading operations including calibrating graders and resolving disputes.
  • Design scope, question types, difficulty levels, and scoring methods for benchmarks.
  • Write and review rubrics to ensure consistent grading.
  • Develop gold-standard answers and test sets.

Skills

Python
Evaluation tools
Benchmark design

Tools

Evaluation frameworks

Job description

Surge AI is seeking a senior evaluator to lead benchmark design and validation for AI research projects in the United States. You will guide grading operations, calibrate graders, and resolve disputes while shaping scoring methods for standardized benchmarks.

Ideal candidates bring 5+ years in evaluation, and proven ability to design assessments and develop gold-standard answers. Proficiency in Python and evaluation tools is required to drive rigorous benchmarking programs.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Evaluation Expert (Fully Remote)
Senior Evaluation Expert (Fully Remote)

Surge AI • United States

Remote
USD 90,000 - 150,000
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera Labs, Inc. • United States

Remote
USD 103,000 - 241,000
Senior AI Evaluation Architect - Remote
Senior AI Evaluation Architect - Remote

Clera Labs, Inc. • United States

Remote
USD 103,000 - 241,000
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • United States

Remote
USD 90,000 - 150,000
Senior Member of Technical Staff, AI Benchmarking
Senior Member of Technical Staff, AI Benchmarking

Artificial Analysis, Inc. • San Francisco (CA)

On-site
USD 100,000 - 150,000
Competitive compensation including equity
Opportunity to shape AI development
Staff AI Benchmark Architect
Staff AI Benchmark Architect

Vals AI, Inc. • San Francisco (CA)

On-site
USD 150,000 - 230,000
Relocation support
Health/dental insurance
Lunch and dinner provided
+2
AI Benchmarks & Evaluations Program Manager
AI Benchmarks & Evaluations Program Manager

Mercor • San Francisco (CA)

On-site
USD 120,000 - 200,000
Performance bonus structure
Equity grant
$15K relocation bonus
+7
Remote Scientific & Technical Evaluation Specialist
Remote Scientific & Technical Evaluation Specialist

Meridial • United States

Remote
USD 14,000 - 41,000
AI Benchmarking & Evaluation Scientist
AI Benchmarking & Evaluation Scientist

snorkelai • United States

Remote
USD 120,000 - 180,000
Senior AI Evaluation Platform Lead
Senior AI Evaluation Platform Lead

EPAM Systems Inc • United States

Remote
USD 140,000 - 190,000