Staff AI Evaluation Scientist

Reflection

San Francisco (CA)

On-site

USD 180,000 - 260,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Top-tier compensation
Stock options
Health & wellness
Meals provided
Parental leave
Unlimited vacation
Visa sponsorship
Team events

Job summary

Reflection AI, Inc. is seeking an evaluation-focused researcher to advance our understanding of model capabilities and to build tight feedback loops between data, evals, and model behavior.

You will collaborate with pre-training, post-training, and applied teams to translate insights into tangible model improvements. The role emphasizes developing generalizable evaluation frameworks for reasoning and alignment, with a bias toward impactful, measurable outcomes in a fast-paced startup setting.

Qualifications

  • Strong statistical analysis and experimental design skills to rigorously measure model improvements.
  • Familiarity with LLM evaluation methodologies: static benchmarks, human preference evals, and/or agentic tasks.
  • High agency and thrive in a fast-paced startup environment; bias for impact over process.
  • Excited to work in a new frontier lab, defining how we measure and accelerate progress toward more capable models.
  • Collaborative, detail-oriented, and motivated by building the feedback loops that make models truly improve.

Responsibilities

  • Conduct critical comparative analysis to advance our understanding of model capabilities.
  • Build and refine evaluation systems and processes that create tight feedback loops between data, evals, and model behavior.
  • Develop generalizable evaluation frameworks that capture what matters for reasoning, alignment, and usefulness.
  • Collaborate closely with pre-training, post-training, and applied teams to translate insights into model improvements.
  • Push the boundaries of what’s measurable, from synthetic evals to human feedback and real-world interaction data.

Skills

Statistical analysis
Experiment design
LLM evaluation
Impact-driven
Start-up environment

Job description

Reflection AI, Inc. is seeking an evaluation-focused researcher to advance our understanding of model capabilities and to build tight feedback loops between data, evals, and model behavior.

You will collaborate with pre-training, post-training, and applied teams to translate insights into tangible model improvements. The role emphasizes developing generalizable evaluation frameworks for reasoning and alignment, with a bias toward impactful, measurable outcomes in a fast-paced startup setting.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Evaluation Engineer
Staff AI Evaluation Engineer

Reflectionai • San Francisco (CA)

On-site
USD 150,000 - 230,000
Stock options
Health & wellness benefits
Meals provided in office
Staff AI Evaluation Engineer for Open Models
Staff AI Evaluation Engineer for Open Models

Reflection AI Ltd • New York (NY)

On-site
USD 120,000 - 200,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff Engineer, ML Evaluation & Model Improvement
Staff Engineer, ML Evaluation & Model Improvement

Reflection AI • San Francisco (CA)

On-site
USD 150,000 - 210,000
Top-tier compensation
Stock options
Health & wellness
Staff Engineer - Evaluation Infrastructure & Research Tools
Staff Engineer - Evaluation Infrastructure & Research Tools

Reflection • New York (NY)

On-site
USD 150,000 - 190,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff ML Evaluation & Data Engineer
Staff ML Evaluation & Data Engineer

Reflection • New York (NY)

On-site
USD 190,000 - 270,000
Stock options
Health & wellness benefits
Daily meals in office
+3
Staff Engineer, Evaluation Infrastructure & ML Systems
Staff Engineer, Evaluation Infrastructure & ML Systems

Reflection AI • San Francisco (CA)

On-site
USD 150,000 - 230,000
Top-tier compensation
Stock options
Health benefits
+5
Senior AI Evaluation Researcher (Remote)
Senior AI Evaluation Researcher (Remote)

ContentBuffer • San Francisco (CA)

Remote
USD 169,000 - 250,000
Equity
Health insurance
Dental coverage
+5
Remote AI QA Architect & Evaluation Lead
Remote AI QA Architect & Evaluation Lead

Meridial • United States

Remote
USD 8,300 - 90,000
Member of Technical Staff - Evaluations
Member of Technical Staff - Evaluations

Reflection AI Ltd • New York (NY)

On-site
USD 120,000 - 200,000
Top-tier compensation
Stock options
Health & wellness
+5
Member of Technical Staff - Evaluations
Member of Technical Staff - Evaluations

Reflectionai • San Francisco (CA)

On-site
USD 150,000 - 230,000
Stock options
Health & wellness benefits
Meals provided in office