Staff AI Evaluation Engineer

Reflectionai

San Francisco (CA)

On-site

USD 150,000 - 230,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Stock options
Health & wellness benefits
Meals provided in office

Job summary

Reflection AI is seeking a results-driven role focused on evaluating and improving large language models. You will design rigorous tests, analyze results, and create scalable evaluation frameworks that inform model development across the company.

You’ll collaborate with cross-functional teams to translate findings into tangible model improvements, balancing rigorous analysis with rapid iteration in a dynamic startup setting.

Qualifications

  • Strong statistical analysis and experimental design skills to rigorously measure model improvements.
  • Familiarity with LLM evaluation methodologies: static benchmarks, human preference evals, and/or agentic tasks.
  • High agency and thrive in a fast-paced startup environment; bias for impact over process.
  • Excited to work in a new frontier lab, defining how we measure and accelerate progress toward more capable models.
  • Collaborative, detail-oriented, and motivated by building feedback loops that improve models.

Responsibilities

  • Conduct critical comparative analysis to advance our understanding of model capabilities.
  • Build and refine evaluation systems and processes that create tight feedback loops between data, evals, and model behavior.
  • Develop generalizable evaluation frameworks that capture what matters for reasoning, alignment, and usefulness.
  • Collaborate closely with pre-training, post-training, and applied teams to translate insights into model improvements.
  • Push the boundaries of what’s measurable, from synthetic evals to human feedback and real-world interaction data.

Skills

Statistical analysis
Experimental design
LLM evaluation methodologies
High agency
Collaborative
Attention to detail

Job description

Reflection AI is seeking a results-driven role focused on evaluating and improving large language models. You will design rigorous tests, analyze results, and create scalable evaluation frameworks that inform model development across the company.

You’ll collaborate with cross-functional teams to translate findings into tangible model improvements, balancing rigorous analysis with rapid iteration in a dynamic startup setting.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Evaluation Engineer for Open Models
Staff AI Evaluation Engineer for Open Models

Reflection AI Ltd • New York (NY)

On-site
USD 120,000 - 200,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff AI Evaluation Scientist
Staff AI Evaluation Scientist

Reflection • San Francisco (CA)

On-site
USD 180,000 - 260,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff Engineer, ML Evaluation & Model Improvement
Staff Engineer, ML Evaluation & Model Improvement

Reflection AI • San Francisco (CA)

On-site
USD 150,000 - 210,000
Top-tier compensation
Stock options
Health & wellness
Staff Engineer - Evaluation Infrastructure & Research Tools
Staff Engineer - Evaluation Infrastructure & Research Tools

Reflection • New York (NY)

On-site
USD 150,000 - 190,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff Engineer - Language Model Evaluation
Staff Engineer - Language Model Evaluation

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 240,000
Shape AI direction
World‑expert in AI
Work with leading AI teams
Staff ML Evaluation & Data Engineer
Staff ML Evaluation & Data Engineer

Reflection • New York (NY)

On-site
USD 190,000 - 270,000
Stock options
Health & wellness benefits
Daily meals in office
+3
Remote AI QA Architect & Evaluation Lead
Remote AI QA Architect & Evaluation Lead

Meridial • United States

Remote
USD 8,300 - 90,000
Senior AI Engineer — RL & LLM Evaluation Lead
Senior AI Engineer — RL & LLM Evaluation Lead

EnableComp LLC • United States

Remote
USD 120,000 - 180,000
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Senior AI/ML Engineer - Remote, Task-Based Evaluations
Senior AI/ML Engineer - Remote, Task-Based Evaluations

prolificacademicltd • United States

Remote
USD 91,000 - 129,000
Remote work
Flexible scheduling