LLM Benchmarking Engineer - On-Site SF/Melbourne

Artificial Analysis

San Francisco (CA)

On-site

USD 120,000 - 170,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Artificial Analysis in San Francisco, CA or Melbourne, Australia, is seeking an engineer to run our LLM benchmarking stack, set up experiments, run benchmarks on language models, analyze results, and own the experimental pipelines. This role is part of our Forward Deployed Engineering team focused on Language Models.

You will collaborate with AI labs on pre-release models and contribute to benchmarking efforts in a fast-paced environment.

Qualifications

  • Responsibilities include setting up experiments
  • Running benchmarks on language models
  • Analyzing results and owning experimental pipelines

Responsibilities

  • Set up experiments and run benchmarks on language models.
  • Analyze results and collaborate with labs on pre-release models.
  • Own and maintain experimental pipelines for benchmarking.

Job description

Artificial Analysis in San Francisco, CA or Melbourne, Australia, is seeking an engineer to run our LLM benchmarking stack, set up experiments, run benchmarks on language models, analyze results, and own the experimental pipelines. This role is part of our Forward Deployed Engineering team focused on Language Models.

You will collaborate with AI labs on pre-release models and contribute to benchmarking efforts in a fast-paced environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Forward Deployed Engineer
Forward Deployed Engineer

Artificial Analysis • San Francisco (CA)

On-site
USD 120,000 - 170,000
LLM Benchmark Engineer — Elevate Model Evaluation
LLM Benchmark Engineer — Elevate Model Evaluation

Vals AI, Inc. • San Francisco (CA)

On-site
USD 180,000 - 240,000
Relocation support
Health insurance
Lunch provided
+3
Frontline Language Model Engineer – AI Benchmarking
Frontline Language Model Engineer – AI Benchmarking

Artificial Analysis • San Francisco (CA)

On-site
USD 140,000 - 190,000
LLM Evaluation Engineer: Benchmark & Infra Lead
LLM Evaluation Engineer: Benchmark & Infra Lead

Nous Research • United States

Remote
USD 120,000 - 180,000
Forward Deployed Engineer, Language Models (Onsite) Equity
Forward Deployed Engineer, Language Models (Onsite) Equity

FDE Times • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity
Senior AI Evaluation Engineer - Remote LLM Benchmarking
Senior AI Evaluation Engineer - Remote LLM Benchmarking

System • New York (NY)

Hybrid
USD 180,000 - 240,000
Meaningful equity ownership
Comprehensive health coverage
Generous paid time off
+1
ML Evaluation Scientist - LLM Benchmarks
ML Evaluation Scientist - LLM Benchmarks

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health coverage
Retirement benefits
L&D stipend
+2
LLM Benchmarking Scientist — Evaluation & Safety
LLM Benchmarking Scientist — Evaluation & Safety

Anyone AI Labs • United States

Remote
USD 120,000 - 180,000
LLM Evaluation Scientist — Benchmarks & Failure Analysis
LLM Evaluation Scientist — Benchmarks & Failure Analysis

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health, dental and vision coverage
Retirement benefits
Learning and development stipend
+3
Applied AI Benchmarking Engineer
Applied AI Benchmarking Engineer

Mercor • San Francisco (CA)

On-site
USD 160,000 - 230,000
Health benefits
Meal stipend
Relocation support
+4