Staff AI Evaluation Engineer for Open Models

Reflection AI Ltd

New York (NY)

On-site

USD 120,000 - 200,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Top-tier compensation
Stock options
Health & wellness
Meals provided in office
Paid parental leave
Unlimited vacation (US) / 30 days (UK)
Visa sponsorship
Team off-sites and celebrations

Job summary

Reflection AI Ltd in New York is seeking a role focused on evaluating and advancing large language models. You will design experiments, build evaluation systems, and translate findings into model improvements, collaborating across pre-training and applied teams.

You have strong statistical design skills, familiarity with LLM evaluation methodologies, and thrive in a fast-paced startup. The role emphasizes impact, collaboration, and delivering measurable progress.

Qualifications

  • Strong statistical analysis and experimental design skills to rigorously measure model improvements.
  • Familiarity with LLM evaluation methodologies: static benchmarks, human preference evals, and/or agentic tasks.
  • High agency and thrive in a fast-paced startup environment; bias for impact over process.
  • Excited to work in a new frontier lab, defining how we measure and accelerate progress toward more capable models.
  • Collaborative, detail-oriented, and motivated by building the feedback loops that make models truly improve.

Responsibilities

  • Conduct critical comparative analysis to advance our understanding of model capabilities.
  • Build and refine evaluation systems and processes that create tight feedback loops between data, evals, and model behavior.
  • Develop generalizable evaluation frameworks that capture what matters for reasoning, alignment, and usefulness.
  • Collaborate closely with pre-training, post-training, and applied teams to translate insights into model improvements.
  • Push the boundaries of what’s measurable, from synthetic evals to human feedback and real-world interaction data.

Skills

Statistical analysis
Experimental design
LLM evaluation
High-ownership
Collaboration

Job description

Reflection AI Ltd in New York is seeking a role focused on evaluating and advancing large language models. You will design experiments, build evaluation systems, and translate findings into model improvements, collaborating across pre-training and applied teams.

You have strong statistical design skills, familiarity with LLM evaluation methodologies, and thrive in a fast-paced startup. The role emphasizes impact, collaboration, and delivering measurable progress.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Evaluation Engineer
Staff AI Evaluation Engineer

Reflectionai • San Francisco (CA)

On-site
USD 150,000 - 230,000
Stock options
Health & wellness benefits
Meals provided in office
Staff AI Evaluation Scientist
Staff AI Evaluation Scientist

Reflection • San Francisco (CA)

On-site
USD 180,000 - 260,000
Top-tier compensation
Stock options
Health & wellness
+5
Staff ML Engineer — Open Models & RL Research
Staff ML Engineer — Open Models & RL Research

Reflection AI Ltd • New York (NY)

On-site
USD 180,000 - 240,000
Top-tier compensation
Stock options
Health & wellness
+5
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Staff Engineer, ML Evaluation & Model Improvement
Staff Engineer, ML Evaluation & Model Improvement

Reflection AI • San Francisco (CA)

On-site
USD 150,000 - 210,000
Top-tier compensation
Stock options
Health & wellness
Senior Research Engineer, AI Evaluation
Senior Research Engineer, AI Evaluation

system • United States

Hybrid
USD 180,000 - 240,000
Equity
Health and wellness coverage
Paid time off
+1
Senior AI Engineer — RL & LLM Evaluation Lead
Senior AI Engineer — RL & LLM Evaluation Lead

EnableComp LLC • United States

Remote
USD 120,000 - 180,000
Staff Engineer - Language Model Evaluation
Staff Engineer - Language Model Evaluation

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 240,000
Shape AI direction
World‑expert in AI
Work with leading AI teams
Senior AI Engineer – RLHF & Model Evaluation (Remote)
Senior AI Engineer – RLHF & Model Evaluation (Remote)

prolificacademicltd • United States

Remote
USD 91,000 - 129,000
Fully remote
Flexible scheduling
Short onboarding
+1
Staff ML Researcher - Open Models & Scalable AI
Staff ML Researcher - Open Models & Scalable AI

Reflection AI • San Francisco (CA)

On-site
USD 210,000 - 300,000
Top-tier compensation
Stock options
Health & wellness
+5