Frontier AI Benchmark Designer (Remote)

HumanitApp

California

Remote

USD 83,000 - 124,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Turing, based in San Francisco, California, is seeking researchers and technical experts to enhance frontier model reasoning by designing and validating benchmark tasks. The role emphasizes building Python-based components, from tasks and environments to tests and golden solutions, and evaluating performance across models on real-world problems.

Applicants should be highly skilled in data science, ML, coding, or frontier STEM disciplines and be able to commit 30+ hours per week.

Qualifications

  • Strong expertise in data science, ML, or coding.
  • Deep quantitative background in frontier STEM.
  • Able to engage reliably for 30+ hrs per week (on weekdays).

Responsibilities

  • Design challenging benchmark tasks for a frontier model.
  • Develop content in a Python-based Agentic environment.
  • Evaluate cross-model performance and identify failures.
  • Analyze agent trajectories to extract core capability loss patterns.
  • Prepare tests and ensure tasks pass evaluation criteria.

Skills

GitHub
Data science
Machine learning
Coding
STEM

Job description

Turing, based in San Francisco, California, is seeking researchers and technical experts to enhance frontier model reasoning by designing and validating benchmark tasks. The role emphasizes building Python-based components, from tasks and environments to tests and golden solutions, and evaluating performance across models on real-world problems.

Applicants should be highly skilled in data science, ML, coding, or frontier STEM disciplines and be able to commit 30+ hours per week.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Frontier AI Benchmark Architect for Reasoning Tasks
Frontier AI Benchmark Architect for Reasoning Tasks

Turing • United States

Remote
USD 150,000 - 230,000
Frontier AI Research Scientist: Benchmarking
Frontier AI Research Scientist: Benchmarking

turing • United States

Remote
USD 250,000 - 400,000
Frontier AI Benchmarking & Red-Teaming Analyst
Frontier AI Benchmarking & Red-Teaming Analyst

HumanitApp • San Francisco (CA)

On-site
USD 83,000 - 138,000
Field Engineer, AI Labs — Frontier Benchmarking
Field Engineer, AI Labs — Frontier Benchmarking

Turing • New York (NY), San Francisco (CA), Seattle (WA)

Hybrid
USD 175,000 - 232,000
Equity
Performance incentives
Comprehensive benefits
Frontier AI Benchmark Engineer
Frontier AI Benchmark Engineer

Handshake • United States

On-site
USD 180,000 - 260,000
Equity
401(k) match
Parental leave
+1
Senior Frontier AI Evaluation Scientist - Equity
Senior Frontier AI Evaluation Scientist - Equity

turing • United States

Remote
USD 250,000 - 350,000
Equity
Publication opportunities
Frontier AI Field Engineer: Benchmarking & Data Delivery
Frontier AI Field Engineer: Benchmarking & Data Delivery

Turing • Oregon

Hybrid
USD 175,000 - 232,000
Frontier AI Research Scientist: Benchmarks and Evaluation
Frontier AI Research Scientist: Benchmarks and Evaluation

turing • United States

Remote
USD 150,000 - 300,000
Frontier AI Evaluation Research Engineer
Frontier AI Evaluation Research Engineer

Cerebro • San Francisco (CA)

On-site
USD 270,000 - 330,000
Equity
Field Engineer, Frontier AI Labs & Benchmarks
Field Engineer, Frontier AI Labs & Benchmarks

FDE Times • New York (NY)

On-site
USD 175,000 - 232,000