Senior AI Evaluation Engineer - Remote LLM Benchmarking

System

New York (NY)

Hybrid

USD 180,000 - 240,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Meaningful equity ownership
Comprehensive health coverage
Generous paid time off
Remote-friendly with Manhattan hub

Job summary

System is seeking a Senior Research Engineer to lead the definition, implementation, and scaling of our LLM evaluation architecture and benchmarking systems. You will contribute to R&D for System’s platform, reporting to the Head of AI & Engineering.

Collaborate with AI researchers, ML infrastructure engineers, and product teams to build automated evaluation loops, create benchmarks and datasets, and mitigate model drift and hallucinations.

Qualifications

  • 3+ years in software engineering or AI research with strong statistical foundations.
  • Experience designing eval harnesses or benchmarking suites for LLMs.
  • Proficient in Python; familiar with PyTorch and Hugging Face.
  • Strong written communication, including papers or blogs.

Responsibilities

  • Design and maintain automated evaluation frameworks for LLM reasoning and retrieval accuracy.
  • Create domain-specific benchmarks, synthetic test-case generators, and golden datasets.
  • Quantify and mitigate edge cases, hallucinations, model drift, and safety regressions.
  • Collaborate with AI researchers, ML infra engineers, and product teams to integrate evaluation loops into CI/CD workflows.

Skills

Software engineering
Applied ML
LLM evaluation
Python

Education

Advanced degree in CS/ML

Tools

PyTorch
Hugging Face
RAG pipelines
Experiment tracking

Job description

System is seeking a Senior Research Engineer to lead the definition, implementation, and scaling of our LLM evaluation architecture and benchmarking systems. You will contribute to R&D for System’s platform, reporting to the Head of AI & Engineering.

Collaborate with AI researchers, ML infrastructure engineers, and product teams to build automated evaluation loops, create benchmarks and datasets, and mitigate model drift and hallucinations.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Model Evaluation Scientist (LLM Benchmarks)
Senior AI Model Evaluation Scientist (LLM Benchmarks)

Cohere • Seattle (WA)

Remote
USD 180,000 - 385,000
Lunch stipend
Health and dental benefits
RRSP matching / 401K / Pension
+5
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Scale AI, Inc. • San Francisco (CA)

On-site
USD 166,000 - 207,000
Health coverage
Equity
Retirement benefits
+3
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc. • New York (NY)

On-site
USD 181,000 - 226,000
Base salary + equity
Health, dental, vision coverage
Retirement benefits
+3
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
ML Evaluation Scientist - LLM Benchmarks
ML Evaluation Scientist - LLM Benchmarks

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health coverage
Retirement benefits
L&D stipend
+2
Senior AI Engineer - Evaluation & LLM Platform (Remote)
Senior AI Engineer - Evaluation & LLM Platform (Remote)

Showcify, Inc. • United States

Remote
USD 180,000 - 260,000
AI Engineer — LLM Production & Evaluation
AI Engineer — LLM Production & Evaluation

LawPro.ai • Virginia

On-site
USD 150,000 - 190,000
Senior AI Engineer: LLM Evaluation, Production & Optimization
Senior AI Engineer: LLM Evaluation, Production & Optimization

LawPro.ai • North Carolina

On-site
USD 140,000 - 190,000
LLM Evaluation Scientist — Benchmarks & Failure Analysis
LLM Evaluation Scientist — Benchmarks & Failure Analysis

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health, dental and vision coverage
Retirement benefits
Learning and development stipend
+3
Senior Legal AI Evaluation Architect (Remote)
Senior Legal AI Evaluation Architect (Remote)

24-Mag Llc • New York (NY)

Remote
USD 400,000 - 800,000
Health insurance
Paid time off
Retirement plan contributions
+1