ML Evaluation Scientist - LLM Benchmarks

Scale AI; Seattle (WA); On-site; USD 181,000 - 226,000; Full time

Benefits offered by this job

Health coverage

Retirement benefits

L&D stipend

Generous PTO

Commuter stipend

Job summary

Scale AI is hiring for roles focused on evaluating and benchmarking frontier LLMs and Agents within the GenAI Research Organization. You will develop robust evaluations and RCA-driven insights, collaborating with researchers to shape evaluation-driven AI development and translate failure analyses into strategic input for next-gen models.

The role emphasizes post-training techniques like SFT and RLHF, publication of findings at major AI conferences, and building benchmarks for text and multimodal

Qualifications

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
• Previous experience in a customer facing role.

Responsibilities

• Analyze model behavior to identify failure modes in frontier LLMs and Agents.
• Design benchmarks and evaluation methods for text and multimodal modalities.
• Apply post-training techniques to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Skills

LLM evaluation

RLHF

SFT

Reward modeling

Diagnostics

Education

PhD or MSc in CS/ML

Tools

Python

PyTorch

Benchmark tools

Job description

Scale AI is hiring for roles focused on evaluating and benchmarking frontier LLMs and Agents within the GenAI Research Organization. You will develop robust evaluations and RCA-driven insights, collaborating with researchers to shape evaluation-driven AI development and translate failure analyses into strategic input for next-gen models.

The role emphasizes post-training techniques like SFT and RLHF, publication of findings at major AI conferences, and building benchmarks for text and multimodal