GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Scale AI; San Francisco (CA); On-site; USD 166,000 - 207,000; Full time

Benefits offered by this job

Health coverage

Equity

Retirement benefits

Learning stipend

Generous PTO

Commuter stipend

Job summary

Scale AI is seeking Research Scientists and Research Engineers to advance evaluation for LLMs and multimodal models. You will analyze failures, design rigorous benchmarks, and connect findings to data and training interventions.

You’ll publish results at top AI conferences and collaborate with leading labs to influence future model development. The role emphasizes strong communication, deep learning expertise, and experience with post-training techniques like RLHF and preference modeling in

Qualifications

• Ph.D. or Master’s degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.

Responsibilities

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents.
• Design and build benchmarks and evaluation methods that measure LLM capabilities in text and multimodal modalities.
• Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Skills

LLM evaluation

RCA (root cause analysis)

Multimodal evaluation

Benchmarks design

Research publication

Communication skills

Education

Ph.D. or Master’s in Computer Science, ML, AI

Job description

Scale AI is seeking Research Scientists and Research Engineers to advance evaluation for LLMs and multimodal models. You will analyze failures, design rigorous benchmarks, and connect findings to data and training interventions.

You’ll publish results at top AI conferences and collaborate with leading labs to influence future model development. The role emphasizes strong communication, deep learning expertise, and experience with post-training techniques like RLHF and preference modeling in