GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc.; New York (NY); On-site; USD 181,000 - 226,000; Full time

Benefits offered by this job

Base salary + equity

Health, dental, vision coverage

Retirement benefits

Learning & development stipend

Generous PTO

Commuter stipend

Job summary

Scale AI, Inc. is recruiting for Research Scientists and Research Engineers specializing in LLM post-training evaluation and benchmark development.

This role focuses on rigorously evaluating frontier models, diagnosing failure modes, and building robust benchmarks for text and multimodal modalities. You will collaborate with researchers and engineers to define best practices in evaluation-driven AI development and translate failure analyses into strategic input for the next generation of

Qualifications

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in machine learning at major conferences and/or journals.
• Previous experience in a customer facing role.

Responsibilities

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents.
• Design and build benchmarks and evaluation methods that measure LLM capabilities in text and multimodal modalities.
• Apply post-training techniques to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Skills

LLM evaluation

SFT RLHF

benchmark development

communication skills

published research

customer facing experience

Education

Ph.D. or Master's in CS/ML/AI

Job description

Scale AI, Inc. is recruiting for Research Scientists and Research Engineers specializing in LLM post-training evaluation and benchmark development.

This role focuses on rigorously evaluating frontier models, diagnosing failure modes, and building robust benchmarks for text and multimodal modalities. You will collaborate with researchers and engineers to define best practices in evaluation-driven AI development and translate failure analyses into strategic input for the next generation of