Staff Engineer - Language Model Evaluation

Artificial Analysis

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Shape AI direction
World‑expert in AI
Work with leading AI teams

Job summary

Artificial Analysis is hiring a Member of Technical Staff to design and scale frontier language model evaluations and benchmarks. You will build datasets, evaluation harnesses and scoring systems, running assessments across major models as they release.

The role blends deep analysis with collaboration across frontier labs and our commercial teams. You will publish results and influence industry understanding of AI capabilities while staying ahead of frontier model developments, leveraging an

Qualifications

  • 3+ years of relevant professional experience across industry or research.
  • Strong analytical and critical thinking skills.
  • Strong Python, with hands-on experience running evaluation harnesses and building datasets.
  • Deep familiarity with the LLM evaluation landscape: benchmarks, contamination, and evaluation methods.
  • Strong statistical grounding: recognizing signal vs noise.
  • Genuine interest and knowledge of Frontier AI.

Responsibilities

  • Design and ship next-generation frontier evaluations across reasoning, knowledge, coding, and agentic capability.
  • Build evaluation datasets, harnesses and scoring systems for contamination-resistant, frontier-scale benchmarks.
  • Shape the future intelligence index by contributing to versions of the AI Index and related platform areas.
  • Publish influential analysis through reports, indexes and data visualizations.
  • Collaborate with frontier labs on pre-release models and coordinate with research teams while business teams manage client relationships.

Skills

Python programming
Analytical thinking
Statistical grounding
Frontier AI interest
Attention to detail

Tools

Evaluation harnesses
Datasets construction
Python

Job description

Artificial Analysis is hiring a Member of Technical Staff to design and scale frontier language model evaluations and benchmarks. You will build datasets, evaluation harnesses and scoring systems, running assessments across major models as they release.

The role blends deep analysis with collaboration across frontier labs and our commercial teams. You will publish results and influence industry understanding of AI capabilities while staying ahead of frontier model developments, leveraging an

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Frontline Language Model Engineer – AI Benchmarking
Frontline Language Model Engineer – AI Benchmarking

Artificial Analysis • San Francisco (CA)

On-site
USD 140,000 - 190,000
Member of Technical Staff (Language Model Evaluations)
Member of Technical Staff (Language Model Evaluations)

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 240,000
Shape AI direction
World‑expert in AI
Work with leading AI teams
Forward Deployed Engineer, Language Models (Onsite) Equity
Forward Deployed Engineer, Language Models (Onsite) Equity

FDE Times • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity
Senior Staff, Applied AI Research - Frontier Evaluations
Senior Staff, Applied AI Research - Frontier Evaluations

FDE Times • San Francisco (CA)

On-site
USD 180,000 - 260,000
Equity
On-site in SF
Staff AI Evaluation Engineer
Staff AI Evaluation Engineer

Reflectionai • San Francisco (CA)

On-site
USD 150,000 - 230,000
Stock options
Health & wellness benefits
Meals provided in office
Staff AI Evaluation Engineer for Open Models
Staff AI Evaluation Engineer for Open Models

Reflection AI Ltd • New York (NY)

On-site
USD 120,000 - 200,000
Top-tier compensation
Stock options
Health & wellness
+5
Remote Frontier AI Evaluation Architect
Remote Frontier AI Evaluation Architect

We Love AI Jobs • San Francisco (CA)

Remote
USD 200,000 - 350,000
Forward Deployed Engineer - Language Models
Forward Deployed Engineer - Language Models

FDE Times • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Forward Deployed Engineer - Language Models
Forward Deployed Engineer - Language Models

Artificial Analysis • San Francisco (CA)

On-site
USD 140,000 - 190,000