Remote AI Evaluation Lead - LLM Benchmarks & Safety

system

United States

Hybrid

USD 180,000 - 240,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Health and wellness coverage
Paid time off
Flexible work arrangement

Job summary

system is seeking an engineer/researcher to lead the design and implementation of evaluation systems for language models used in clinical decision support. You will combine applied research and engineering to measure model reasoning, retrieval, and structured responses, and ensure quality and safety checks are reproducible across development and deployment.

You will build evaluation frameworks, develop benchmarks, test cases, and reference datasets, and address edge cases, hallucinations, drift,

Qualifications

  • Three+ years in software engineering, applied machine learning, or AI research with strong statistics.
  • Experience building LLM evaluation harnesses, benchmarks, or reference metrics.
  • Strong technical writing and communication skills.

Responsibilities

  • Build and maintain automated evaluation frameworks for model outputs.
  • Develop general and domain-specific benchmarks, synthetic test cases, and trusted reference datasets.
  • Measure edge cases, hallucinations, model drift, and safety regressions.
  • Integrate evaluation practices into CI/CD workflows with research, infra, and product partners.

Skills

Python
Statistics
Scientific rigor

Education

Advanced degree in computer science / machine learning / statistics

Tools

PyTorch
Hugging Face
Model APIs
Retrieval-augmented generation
Experiment tracking / observability tools

Job description

system is seeking an engineer/researcher to lead the design and implementation of evaluation systems for language models used in clinical decision support. You will combine applied research and engineering to measure model reasoning, retrieval, and structured responses, and ensure quality and safety checks are reproducible across development and deployment.

You will build evaluation frameworks, develop benchmarks, test cases, and reference datasets, and address edge cases, hallucinations, drift,

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Evaluation Engineer - Remote LLM Benchmarking
Senior AI Evaluation Engineer - Remote LLM Benchmarking

System • New York (NY)

Hybrid
USD 180,000 - 240,000
Meaningful equity ownership
Comprehensive health coverage
Generous paid time off
+1
AI Evaluation Architect
AI Evaluation Architect

Nouvells • United States

Remote
USD 90,000 - 130,000
LLM Benchmarking Scientist — Evaluation & Safety
LLM Benchmarking Scientist — Evaluation & Safety

Anyone AI Labs • United States

Remote
USD 120,000 - 180,000
Senior Research Engineer, AI Evaluation
Senior Research Engineer, AI Evaluation

system • United States

Hybrid
USD 180,000 - 240,000
Equity
Health and wellness coverage
Paid time off
+1
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Scale AI, Inc. • San Francisco (CA)

On-site
USD 166,000 - 207,000
Health coverage
Equity
Retirement benefits
+3
Senior AI Engineer — RL & LLM Evaluation Lead
Senior AI Engineer — RL & LLM Evaluation Lead

EnableComp LLC • United States

Remote
USD 120,000 - 180,000
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc. • New York (NY)

On-site
USD 181,000 - 226,000
Base salary + equity
Health, dental, vision coverage
Retirement benefits
+3
Senior AI/ML Engineer - Remote LLM Training & Evaluation
Senior AI/ML Engineer - Remote LLM Training & Evaluation

prolificacademicltd • United States

Remote
USD 66,000 - 110,000
Fully remote
Hourly tasks paid up to $80
Flexible contributor arrangement
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Senior AI Evaluation & Benchmarking Lead
Senior AI Evaluation & Benchmarking Lead

TRSS • Ann Arbor (MI)

Hybrid
USD 127,000 - 237,000
Hybrid Work Model
Mental Health Days
Headspace app
+2