Staff AI Evaluation Engineer for Open Models

Reflection AI Ltd; New York (NY); On-site; USD 120,000 - 200,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided in office

Paid parental leave

Unlimited vacation (US) / 30 days (UK)

Visa sponsorship

Team off-sites and celebrations

Job summary

Reflection AI Ltd in New York is seeking a role focused on evaluating and advancing large language models. You will design experiments, build evaluation systems, and translate findings into model improvements, collaborating across pre-training and applied teams.

You have strong statistical design skills, familiarity with LLM evaluation methodologies, and thrive in a fast-paced startup. The role emphasizes impact, collaboration, and delivering measurable progress.

Qualifications

• Strong statistical analysis and experimental design skills to rigorously measure model improvements.
• Familiarity with LLM evaluation methodologies: static benchmarks, human preference evals, and/or agentic tasks.
• High agency and thrive in a fast-paced startup environment; bias for impact over process.
• Excited to work in a new frontier lab, defining how we measure and accelerate progress toward more capable models.
• Collaborative, detail-oriented, and motivated by building the feedback loops that make models truly improve.

Responsibilities

• Conduct critical comparative analysis to advance our understanding of model capabilities.
• Build and refine evaluation systems and processes that create tight feedback loops between data, evals, and model behavior.
• Develop generalizable evaluation frameworks that capture what matters for reasoning, alignment, and usefulness.
• Collaborate closely with pre-training, post-training, and applied teams to translate insights into model improvements.
• Push the boundaries of what’s measurable, from synthetic evals to human feedback and real-world interaction data.

Skills

Statistical analysis

Experimental design

LLM evaluation

High-ownership

Collaboration

Job description

Reflection AI Ltd in New York is seeking a role focused on evaluating and advancing large language models. You will design experiments, build evaluation systems, and translate findings into model improvements, collaborating across pre-training and applied teams.

You have strong statistical design skills, familiarity with LLM evaluation methodologies, and thrive in a fast-paced startup. The role emphasizes impact, collaboration, and delivering measurable progress.